AWS Database Blog

Import Amazon RDS Enhanced Monitoring metrics into Amazon CloudWatch

Database administrators, application teams, and architects want to proactively address performance issues before they affect the end-user experience or cause an outage. To achieve this, you need the right level of visibility into database health indicators. Amazon Relational Database Service (Amazon RDS) provides different monitoring tools like Amazon RDS Enhanced Monitoring and Amazon RDS Performance Insights, which you can use in addition to the default Amazon CloudWatch Metrics published by the RDS instance.

Enhanced Monitoring is a feature that you can add to your RDS instance and get granular operating system (OS) metrics. While working with RDS, you may need to do computations on Enhanced Monitoring metrics, compare the metrics by overlaying one on top of another, and create alarms on the metrics.

In this post, we show you how to import multiple Enhanced Monitoring metrics to CloudWatch and use the full capabilities of CloudWatch on those metrics.

Solution overview

You can view Enhanced Monitoring metrics on the Amazon RDS console. They are stored in CloudWatch logs under the log group RDSOSMetrics. With the solution provided in this post, you could do the following:

  • View metrics for more than 1 hour
  • House multiple metrics in the same time series graph
  • Edit the granularity of the metric data points (more than 60 seconds)
  • Swap axes for comparison
  • Do mathematical computation

Performance Insights has a custom dashboard that allows you to view and co-relate Enhanced Monitoring metrics with other metrics. Performance Insights provides this feature in its free tier. The solution presented in this post incurs additional costs to view Enhanced Monitoring metrics in CloudWatch. The estimated charges per month includes $3.00 per dashboard and $0.3 per every metric extracted.

The following diagram illustrates the solution architecture.

We use an AWS CloudFormation template to automate the import of multiple Enhanced Monitoring metrics of an RDS instance to CloudWatch. The stack also creates a CloudWatch dashboard with metrics grouped by category (like CPU and memory) for a user-friendly viewing experience. This is critical when comparative analysis is required between multiple metrics of the same RDS instance or even across multiple RDS instances for performance troubleshooting or behavior pattern identification.

The CloudWatch dashboard hosts all the Enhanced Monitoring metrics specified in the CloudFormation stack parameters. Any user can visualize the metrics in the CloudWatch dashboard as they would with standard RDS CloudWatch metrics.

Prerequisites

In this post, we deploy the CloudFormation template and export Enhanced Monitoring metrics from an Amazon RDS for MySQL instance to CloudWatch. To achieve this, complete the following prerequisites:

  1. Create an RDS for MySQL instance.
  2. Enable Enhanced Monitoring on the RDS instance.

CloudFormation template script

Save this script either on your local machine or Amazon Simple Storage Service (Amazon S3):

Deploy the CloudFormation stack

To deploy the stack, complete the following steps:

  1. On the AWS CloudFormation console, choose Create stack and choose With new resources (Standard).
  2. Choose Upload a template file.
  3. Choose Choose file and browse to the location where the template is stored.
  4. Choose Next.
  5. For Stack name, enter a name (for example, RDSOS).
  6. Under Database Engine, choose SQL Server or Other Engine (Select based on the engine type).
  7. Under RDS OS metrics, keep the parameters set to true unless they’re not required, in which case you can set them to false.
  8. For DatabaseInstanceID, enter the RDS database identifier.
  9. For Namespace, enter a namespace (for example, RDSOS).
  10. For Dashboard, enter a name for the dashboard (for example, RDSOS-Dashboard).
  11. Choose Next.
  12. Choose Next again.
  13. Choose Submit.

The stack creates metric filters on the log group RDSOSMetrics and filters on the RDS identifier that you provided. If you have a separate automation with the metrics from RDSOSMetrics as the source, that will not be impacted because of this stack creation.

You can reuse the same CloudFormation template for other RDS instances as well.

You can track the creation of the related resources on the Resources tab of the stack. When all the resources are in CREATE_COMPLETE state after 10minutes, you can visit the CloudWatch console and go to the Dashboards page and locate the dashboard created.

Alternatively, you can view the metrics on the Metrics page.

Considerations

The solution has the following considerations.

  • This stack creates metrics for CPUUtilization (for all engines), memory (for all engines), LoadavgMinute (for all engines except SQL Server), Tasks (for all engines except SQL Server), System (for SQL Server) and Swap (for all engines except SQL Server). Disk I/O and throughput metrics are not extracted via this script.
  • You can deploy the stack for Amazon Aurora MySQL-Compatible Edition and Amazon Aurora PostgreSQL-Compatible Edition instances, but not on the cluster.
  • Each stack deployment can create a maximum of 37 metrics for any RDS engine except Amazon RDS for SQL Server. It creates 17 metrics when the source engine is Amazon RDS for SQL Server.
  • You delete the stack, all the associated resources created by the stack (metric filters and dashboards) are deleted along with it.
  • You can create a maximum of 100 metric filters per AWS Region. This is a hard limit and cannot be modified by AWS Support.
  • CloudWatch custom dashboards and custom metric filters are chargeable. For more information, refer to Amazon CloudWatch Pricing.
  • The metrics in CloudWatch will start to populate from the time the stack deployment is complete.

Note – since this solution uses metric filters and there is a hard limitation of a 100 metric filters per AWS Region, we recommend you to add the metrics which are critical for monitoring and only for critical instances. You can opt to choose an instance, deploy the solution and then delete the stack once the monitoring needs are done and thereby freeing up the metric filter usage. Example – Metrics like cpu:nice, cpu:user and memory:swap can give you insights on your CPU and memory usage. Refer to Monitor real-time Amazon RDS OS metrics with flexible granularity using Enhanced Monitoring for more details on Enhanced Monitoring metrics

Clean up

If you no longer need to use this solution, complete the following steps to clean up your resources:

  1. Delete the RDS instance.
  2. Delete the CloudWatch log group.
  3. Delete the CloudFormation stack.

Conclusion

In this post, you learned how to export multiple Enhanced Monitoring metrics to CloudWatch using CloudFormation template which creates the metric filters and the dashboard. After the metrics are available in CloudWatch, you can also create CloudWatch alarms to monitor them, using the full potential of the CloudWatch, or delete them with no impact to the RDS instance.

If you have any questions or comments, leave them in the comments section.

About the Authors

Sameer Kumar is a Principal Database Specialist TAM at Amazon Web Services. He focuses on Amazon RDS, Amazon Aurora , and Amazon DocumentDB. He works with enterprise customers providing technical assistance on database operational performance and sharing database best practices.

Deepak Sharma is a Database Specialist TAM at Amazon Web Services focusing on Amazon RDS PostgreSQL, RDS Oracle, and Amazon Aurora engines. He works with enterprise customers to provide technical assistance on operational performance, help them run optimized workloads on AWS, and share database best practices.

Nirupam Datta is a Senior Cloud Support DBE at AWS. He has been with AWS for over 4 years. With over 12 years of experience in database engineering and infra-architecture, Nirupam is also a subject matter expert in the Amazon RDS core systems and Amazon RDS for SQL Server. He provides technical assistance to customers, guiding them to migrate, optimize, and navigate their journey in the AWS Cloud.

Chaitanya Gummadi is a Sr Cloud Support Engineer at AWS. He’s passionate about helping customers solve their AWS issues in the Networking domain. When not helping customers, he likes to explore different cuisines and go hiking.