How can I filter Enhanced Monitoring CloudWatch Logs to generate automated custom metrics for Amazon Relational Database Service (Amazon RDS)?

Enhanced Monitoring for RDS provides the following OS metrics:

  • Free Memory
  • Active Memory
  • Swap Free
  • Processes Running
  • File System Used

You can use these metrics to understand the environment's performance, and these metrics are ingested by Amazon CloudWatch Logs as log entries. You can use CloudWatch to create alarms based on metrics. These alarms execute actions, and you can publish these metrics from within your infrastructure, device, or application into CloudWatch as a custom metric. By using Enhanced Monitoring and CloudWatch together, you can automate tasks by creating a custom metric for the CloudWatch Logs RDS ingested date from the Enhanced Monitoring metrics.

Create a custom metric using filters on a log group

Note: These steps require that Enhanced Monitoring is enabled in your RDS instance. For more information, see Monitoring Amazon RDS.

1.    Enable Enhanced Monitoring.

2.    Open the Amazon CloudWatch console, and then choose Logs from the navigation pane.

3.    Filter for RDSOSMetrics from the list of Log Groups, and choose the value listed under the Metric Filters column.

4.    Choose Create Metric Filter, and choose a Filter Pattern term for your RDS instance. For example:

DISK UTILIZATION: { $.diskIO[0].writeKbPS="0.00"}
DISK WRITE KBPS : { $.diskIO[0].util="0.00"}
DISK READ KBPS : { $.diskIO[0].readKbPS="0.00"}
DISK WRITE IOPS : { $.diskIO[0].writeKbPS="0.00"}

Note: Be sure to update 0.00 to the value that you want to measure.

5.    Select the Log Data to test, and then choose Test Pattern.

6.    Choose Assign Metric.

7.    Enter a Filter Name, select a Metric Namespace and Metric Name, and then choose Show advanced metric settings.

8.    Enter a Metric Value—for example, $.cpuUtilization.idle—and then choose Create Filter.

Repeat these steps for each additional metric that you add.

Automation

There are more than 60 monitoring metrics per RDS instance, and you can use a script to automate this process. See the awslabs rds-support-tools repository for an example script that uses an RDS for MySQL DB instance that works with the following engines:

  • MySQL
  • MariaDB
  • Amazon Aurora (MySQL and PostgreSQL)
  • PostgreSQL

Specify the RDS instance that has Enhanced Monitoring enabled and the namespace where you want these metrics to reside—optionally, you can add the names of the metrics and the Region. If no optional fields are specified, the script will consider all the metrics to publish, and then use the Region that is specified by default in the file .aws/config using the AWS Command Line Interface (AWS CLI).

Note: The script does not create metrics for "process list," so you might need to create filters manually, depending on which process you want to display.

The names of the metrics must be specified by the following pattern:

group.metricname

The following sample code illustrates a call to enable this metric for cpuUtilization.idle and diskIO.readKbPS:

python RDSCreateMetricsFromEnhancedMonitoring.py --rds_instance mysqltest --namespace MySQL --metrics_to_filter cpuUtilization.idle diskIO.readKb

You can then create alarms for these custom metrics. For more information, see Creating Amazon CloudWatch Alarms.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2017-01-09

Updated: 2018-09-25