AWS Cloud Operations Blog
Creating a near-realtime dashboard on Amazon CloudWatch for a Migration usecase
Monitoring performance metrics of AWS resources is crucial for any business use case running in cloud. AWS Well-Architected Framework best practices recommend customers to setup observability and monitoring for their infrastructure on AWS. But, as AWS usage for customer increases, it gets harder to observe and visualize the data and metrics at scale and identify any performance blockers. AWS provides readily available Amazon CloudWatch metrics for AWS resources that are system-defined. Customers can also create custom metrics on Amazon CloudWatch that are user-defined. But over the years, customers have found it hard to visualize all these metrics together at one place without the right visualization tool.
In this blogpost, you will learn how to visualize metrics for several resources on a single Amazon CloudWatch dashboard and make it scalable. We will create a near-realtime dashboard on Amazon CloudWatch with Migration as the use case. The user-defined metrics will be derived from the AWS service used for Migration. In this blog, we discuss how to automatically put and update the system-defined and user-defined CloudWatch metrics on the Amazon CloudWatch dashboard.
Architecture
Figure 1. Architecture diagram for creating a near-realtime Amazon CloudWatch Dashboard
This is the architectural diagram for creating a near-realtime dashboard on Amazon CloudWatch for Migration use case. In this blog, we will be monitoring the AWS Application Migration Service (AWS MGN). AWS MGN is a highly automated move and improve migration service that can migrate physical, virtual, or any cloud servers “as is” to the AWS cloud.
1. The solution above starts with an Amazon EventBridge cron rule that runs every 1 minute to invoke the AWS Lambda function.
2. The AWS Lambda function will run and gather Amazon CloudWatch metrics from AWS MGN for monitoring.
3. The AWS Lambda function will then publish these metrics on to a Amazon CloudWatch dashboard. This dashboard will be used by the viewer to see performance metrics or other statistics. The red dotted box in the architecture diagram above can be generalized for any use case.
Solution Overview
1. The Lambda code here in python is customized for the Migration use case. You will be using the following code in this lambda function to push system-defined and user-defined metrics to Amazon CloudWatch dashboard.
Note – The lambda function is compatible with python 3.0 or above. This function is designed to receive metrics from AWS MGN and create custom CloudWatch metrics from it. It is designed to create test, cutover and replication state custom metrics and push them on to Amazon CloudWatch. Then, the lambda function is creating an Amazon CloudWatch dashboard and including system-defined metrics provided by AWS MGN service and user-defined custom metrics generated by the Lambda function itself. If you are using the above code as is, then please make sure to replace <region> with a specific AWS region such as us-east-1, or us-west-2 etc.
2. Zip the above Lambda function or the custom Lambda function for your use case and upload it to an S3 bucket of your choice. Name the zip file as “my-deployment-package.zip”. Use this link to learn how to zip a python file.
3. Use the YAML template here to launch the infrastructure through AWS Cloudformation in the AWS region selected in the step 1. In this YAML template, replace <s3-bucket> with the s3 bucket name derived from step 2. Deploy this YAML template through AWS Cloudformation, and it will launch the Lambda function, AWS Event Bridge rule and necessary Lambda roles and permissions.
Make sure that the AWS region you choose to deploy the Cloudformation YAML template matches the region in your Lambda function code and the S3 bucket from step 2.
Note that the YAML template creates a Lambda role with the policy that is required to access the AWS MGN service.
4. Once the environment is deployed by AWS CloudFormation, go to Find Services and search for CloudWatch.
5. On the left navigation pane choose Dashboards, then choose MGN-Dashboard. If you have source machines added in the AWS MGN console, then you will see 5 widgets in total. We will add one more in the following steps.
6. In the top right corner, choose the + (plus) button to add a new widget.
7. Choose Number.
8. Search for mgn in the search bar and choose AWS MGN under Custom namespaces.
9. Choose Across all Source Servers.
10. Select the first two metrics: MachinesFullySynced and MachinesSyncing. Then select the Graphed metrics tab. Change the statistic for both the metrics to Maximum and Period to 1 second as shown below. Select Create Widget.
11. Similarly, you can add other custom metrics (generated by the lambda function) as widgets to the Amazon CloudWatch dashboard. You can add more metrics to this dashboard after it is created which makes it scalable. The Eventbridge rule runs every 1 min to update this dashboard which also makes it near-realtime.
12. Set the duration for events displayed on dashboard to Custom (5m) interval instead of default (3h) so that dashboard only shows data for all metrics during the last 5 minutes.
13. Choose Save dashboard.
Result
The CloudWatch dashboard is near-realtime that has all the metrics that you added for visualizing the status of your Migration. This CloudWatch dashboard is also scalable as you can continue to add more widgets/metrics without making any changes to the existing
settings.
Conclusion
We saw how easy it is to create a near-realtime dashboard on Amazon CloudWatch and make it scalable for adding more metrics to it. Most system-defined CloudWatch metrics are updated automatically but the solution discussed in this blogpost can also be used for periodically updating the user-defined metrics on the Amazon CloudWatch dashboard. Moreover, you can reuse this solution and some of the existing code for other non-migration use cases as well.
The Cloudwatch charges will be minimal for this scenario but please refer to the CloudWatch pricing page for custom use cases.