Use AWS Systems Manager to enable CloudWatch memory metrics for Windows Server Amazon EC2 instances

In this blog post, we will show how to reduce the administrative burden of enabling Amazon CloudWatch memory metric monitoring on Windows Server EC2 instances using AWS Systems Manager automation. Once enabled, we will show how memory monitoring benefits downstream services like AWS Compute Optimizer to provide accurate cost savings recommendations.

Monitoring is an important part of maintaining the reliability, availability, and performance of your Amazon Elastic Compute Cloud (Amazon EC2) instances and your AWS solutions. CloudWatch monitors workloads you run on AWS in near real-time. CloudWatch allows you to collect and track metrics, which are variables you can measure for your resources and applications. By default, CloudWatch can monitor Amazon EC2 system metrics for CPU, storage, and networking. For a complete list of CloudWatch metrics see the documentation, List the available CloudWatch metrics for your instances.

Memory metrics are important to system monitoring, especially for memory-intensive Microsoft applications like SQL Server and Exchange Server. In addition, customers can improve their cost savings by 327% on average using AWS Compute Optimizer recommendations by installing the Amazon CloudWatch agent on their EC2 instances to collect memory utilization metrics. Enabling memory metric monitoring for CloudWatch requires installing the CloudWatch agent and defining additional custom metric configurations.

Solution overview

In this blog post, we use an AWS CloudFormation template to perform the following actions.

1. Assume an AWS Identity and Access Management (IAM) role to allow Systems Manager to run a Systems Manager Automation runbook.

For Systems Manager to perform actions on an Amazon EC2 instance, it must have the appropriate permissions. IAM roles are used to perform these types of actions. The CloudFormation template uses the role AmazonSSMAutomationRole, which is a default role managed by AWS.

2. Configure the CloudWatch agent to collect memory metrics.

The CloudWatch agent is used to collect metrics from Amazon EC2 instances and report those metrics to the CloudWatch service. Before running the CloudWatch agent on any servers, you must create one or more CloudWatch agent configuration files. The agent configuration file is a JSON file that specifies the metrics, logs, and traces that the agent is to collect, including custom metrics.

The CloudFormation template creates the CloudWatch agent configuration file in the Parameter Store, a feature of Systems Manager which provides secure, hierarchical storage for configuration data management and secrets management. The configuration specifies to capture Windows Server performance metrics Available MBytes and % Committed Bytes In Use from the Windows Server EC2 instances.

• Available Mbytes, a performance counter in Windows that indicates the amount of physical memory (RAM) available for allocation to processes or the operating system. It represents the pool of memory that is currently unused and can be assigned to new or existing processes upon request.
• % Committed Bytes In Use, a performance counter in Windows that measures the amount of virtual memory currently being used by processes and the operating system itself. It is an important metric for workloads like SQL Server because it helps identify potential memory pressure or excessive memory usage, which can impact overall system performance.

Linux operating systems use different memory performance metrics, this configuration is only valid for Windows Servers. See Setup memory metrics for Amazon EC2 instances using AWS Systems Manager if you need to deploy the CloudWatch agent and enable memory metrics for Amazon EC2 Linux instances.

All custom CloudWatch metrics charges are prorated by the hour and metered only when metrics are sent to CloudWatch. Visit the CloudWatch pricing page for additional pricing details.

3. Create an Automation runbook to install and configure the Amazon CloudWatch agent.

An Automation runbook defines the actions Systems Manager performs on your managed instances. The CloudFormation template creates an Automation runbook performing the following actions.

• Attach the AWS Managed Policy CloudWatchAgentServerPolicy to the IAM instance profile.
• Install the CloudWatch agent.
• Apply the CloudWatch agent memory metrics configuration (Step 2).

4. Run a Systems Manager Association to target all Windows Server Amazon EC2 instances reporting to Systems Manager.

The final step is to create a Systems Manager Association, which specifies the targets and schedule for the Systems Manager Automation runbook. The CloudFormation template is set to target all instances which are reporting as Managed Nodes in Systems Manager Fleet Manager.

Prerequisites

The CloudFormation template will only work on Windows Server Amazon EC2 instances which are reporting to Systems Manager. Verify any servers you want to deploy the CloudWatch agent are online and displaying in AWS Systems Manager Fleet Manager as a Managed Node. For information on how to setup Systems Manager for Amazon EC2 instances, please review, Setting up Systems Manager for EC2 instances .

Walkthrough

Deploy the CloudFormation Stack

1. Download the CloudfFormation file mem-metrics-windows.yml.
2. In the AWS Management Console open the CloudFormation console.
3. In the navigation pane, choose Stacks.
4. Choose Create stack, choose With existing resources (import resources), and choose Next.
5. For Template resource, select Upload a template file.
6. Select Choose file and then upload the mem-metrics-windows.yml file, choose Next.
7. On the Identify resources page, choose Next.
8. On the Specify stack details page, for Stack name, enter a name for your stack, and then choose Next.
9. Review your stack, then choose Import resources.

Verify the Association

1. After the stack is deployed, choose the Resources tab.
2. Copy the Physical ID for the resource type AWS::SSM::Association (Figure 1).
   

   Figure 1. CloudFormation – Find the Physical ID for the Systems Manager Association.

3. Open the Systems Manager console.
4. In the navigation pane, under Node Management, choose State Manager.
5.  On the Associations page, select the Association id which matches the Physical ID copied from the Stack Resources.
6. Filter Associations, choose Association id, paste your Stack Resource Physical ID (Figure 2).
   

   Figure 2. Systems Manager – Find the Association id by filtering for the Physical ID.

7. Choose the Execution history
8. Selecting the Execution ID will display the execution targets (Amazon EC2 instances) (Figure 3).
   

   Figure 3. Systems Manager – review the Association execution targets.

(Optional) Verify the CloudWatch agent installation on the Windows Server

1. Log into your Windows Server Amazon EC2 instance.
2. Open Event Viewer (eventvwr.msc).
3. Under Windows Logs, choose Application.
4. Review Event ID 1040 to see the beginning of the CloudWatch agent installation (Figure 4).
   

   Figure 4. Windows Server Event Log – starting the CloudWatch agent installation.

5. Review Event ID 1033 to check the CloudWatch agent installation has completed successfully (Figure 5).
   

   Figure 5. Windows Server Event Log – Successful installation of the CloudWatch agent.

Verify memory metrics are reporting to CloudWatch.

1. Open the CloudWatch console.
2. In the Navigation Pane, browse Metrics, then choose All metrics.
3. On the Browse tab, under Custom Namespaces select CWAgent.
4. Select InstanceId.
5. Find your Instance by the InstanceId.
6. Verify Memory % Committed Bytes in Use and Memory Available Bytes metrics (Figure 6).
   

   Figure 6. CloudWatch – Verify memory metrics.

Review Compute Optimizer recommendations.

AWS Compute Optimizer is a service that analyzes your AWS resources’ configuration and utilization metrics to provide you with rightsizing recommendations. It reports whether your resources are optimal and generates optimization recommendations to reduce the cost and improve the performance of your workloads.

A major benefit to enabling CloudWatch memory monitoring is unlocking the ability for Compute Optimizer to provide Amazon EC2 recommendations based on memory usage. Without the use of the CloudWatch agent, Compute Optimizer will not have access to memory metrics and will only provide recommendations based on CPU, storage, and networking utilization.

1. Open the Compute Optimizer console.
2. In the Navigation pane, select EC2 Instances.
3. Select an EC2 instance using the InstanceId.
4. Verify memory utilization when comparing recommendations (Figure 7).
   

   Figure 7. Compute Optimizer – verify memory utilization.

Note: It can take up to 30 hours for Compute Optimizer to begin providing recommendations.

Cleanup

Deploying this solution will provision AWS resources and incur cost. Once you have completed testing and you no longer need to collect metrics on new instances, you can remove the provisioned resources by deleting the AWS CloudFormation stack you created as a part of this example. Navigate to the CloudFormation Console and select the stack that you created. Choose Delete, and then choose Confirm when prompted. To uninstall the unified CloudWatch agent, follow the steps, Uninstall a package and select AmazonCloudWatchAgent as a name for the package. Then detach CloudWatchAgentServerPolicy from IAM instance profile(s), follow the steps in, Removing IAM identity permissions (console).

Conclusion

In this blog post, we learned how to configure CloudWatch memory metrics on Windows Server EC2 instances using AWS Systems Manager. We then reviewed how memory metric monitoring can help save costs by increasing the accuracy of AWS Compute Optimizer recommendations.

AWS has significantly more services, and more features within those services, than any other cloud provider, making it faster, easier, and more cost effective to move your existing applications to the cloud and build nearly anything you can imagine. Give your Microsoft applications the infrastructure they need to drive the business outcomes you want. Visit our .NET on AWS and AWS Database blogs for additional guidance and options for your Microsoft workloads. Contact us to start your migration and modernization journey today.