How to create an Amazon EC2 Auto Scaling policy based on a memory utilization metric (Linux)

This is the first in a two-part series about how to create an Amazon EC2 Auto Scaling policy based on memory utilization metric. This post covers Linux OS. In part 2 I’ll cover how to create an Amazon EC2 Auto Scaling policy based on a memory utilization metric in Windows OS.

Amazon EC2 Auto Scaling helps you maintain application availability and allows you to automatically add or remove EC2 instances according to conditions you define. You can use the fleet management features of EC2 Auto Scaling to maintain the health and availability of your fleet. You can also use the dynamic and predictive scaling features of EC2 Auto Scaling to add or remove EC2 instances. Dynamic scaling responds to changing demand. Predictive scaling schedules the right number of EC2 instances based on predicted demand. You can use dynamic scaling and predictive scaling together to scale faster.

Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. CloudWatch provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. You can use CloudWatch to detect anomalous behavior in your environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep your applications running smoothly.

You can use the unified CloudWatch agent to do the following:

Collect internal system-level metrics from EC2 instances across operating systems. The metrics can include in-guest metrics, in addition to the metrics for EC2 instances. For more information, see Metrics Collected by the CloudWatch Agent in the Amazon CloudWatch User Guide.
Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment and servers that are not managed by AWS.
Retrieve custom metrics from your applications or services using the StatsD and collectd protocols. StatsD is supported on both Linux servers and servers running Windows Server. collectd is supported on Linux servers only.
Collect logs from EC2 instances and on-premises servers running Linux or Windows Server.

In this post, I’ll show you how to create an Amazon EC2 Auto Scaling policy based on custom metrics such as the memory utilization metric for a Linux fleet.

Overview

A scaling policy instructs Amazon EC2 Auto Scaling to track a specific CloudWatch metric. It defines the action to take when the associated CloudWatch alarm is in an ALARM state. The metrics that are used to trigger an alarm are an aggregation of metrics coming from all of the instances in the Auto Scaling group. For example, let’s say you have an Auto Scaling group with two instances where one instance is at 60 percent CPU and the other is at 40 percent CPU. On average, they are at 50 percent CPU. When the policy is in effect, Amazon EC2 Auto Scaling adjusts the group’s desired capacity up or down when the alarm is triggered.

Amazon EC2 Auto Scaling supports target tracking scaling, step scaling, and simple scaling. In a target tracking scaling policy, you can use predefined or customized metrics. You can choose other metrics available in Amazon CloudWatch or you can specify customized metrics in CloudWatch. You must use the AWS CLI or an SDK to create a target tracking policy with a customized metric.

Keep the CustomizedScalingMetricSpecification in mind when you choose a metric:

The metric must be a valid utilization metric and describe how busy an instance is.
The metric value must increase or decrease proportionally to the number of instances in the Auto Scaling group. For example, the CPU utilization of an Auto Scaling group works (that is, the Amazon EC2 metric CPUUtilization with the metric dimension AutoScalingGroupName) if the load on the Auto Scaling group is distributed across the instances.

The memory utilization metric is not one of the metrics that Amazon EC2 sends to CloudWatch. I will use the unified CloudWatch agent to send the memory utilization metric from EC2 instances. I will also configure the aggregation_dimensions field in the CloudWatch agent configuration file. This field specifies the dimensions that collected metrics are to be aggregated on. For example, if you roll up metrics on the AutoScalingGroupName dimension, the metrics from all instances in each Auto Scaling group are aggregated and can be viewed as a whole.

I will use a launch template to create an Amazon EC2 Auto Scaling group that installs the CloudWatch unified agent and configures it with custom metrics aggregated on the AutoScalingGroupName dimension using the AWS Systems Manager Parameter Store. Then I’ll create a scaling policy on the CloudWatch agent custom metric.

Amazon EC2 launch template with user data to install the CloudWatch data and configure the agent using the Systems Manager Parameter Store. The scaling policy is based on the CloudWatch custom metric alarm triggering.

Figure 1: Solution flow

Walkthrough

In this walkthrough, you will complete the following steps:

Create Systems Manager parameter with the CloudWatch agent configuration to create an aggregated metric on memory usage percentage.
Create an AWS Identity and Access Management (IAM) role to use with the CloudWatch agent.
Install and configure the CloudWatch unified agent on your Auto Scaling fleet. I use Amazon EC2 user data in the Amazon EC2 Auto Scaling launch templates with AWS Systems Manager Parameter Store.
Create an Amazon EC2 Auto Scaling group.
Create an Amazon EC2 Auto Scaling policy that uses target tracking scaling policies or step scaling policies.

You can use the AWS CloudFormation template provided in this post to:

Create a Systems Manager parameter.
Create an Amazon EC2 Auto Scaling launch template with the latest Amazon Linux 2 AMI.
Create an Amazon EC2 Auto Scaling group and target tracking policy on your Linux fleet.

Prerequisites

You need an AWS account with the IAM permissions required to access Amazon CloudWatch, Amazon EC2, Amazon EC2 Auto Scaling, and AWS Systems Manager.

Create Systems Manager Parameter

I configured the agent to aggregate the mem_used_percent metric on the AutoScalingGroupName dimension. I also configured the agent to send the mem_used_percent metric for each instance individually with the InstanceId dimension. If you don’t want to send the mem_used_percent metric for each instance, then remove the append dimensions for InstanceId.

Note that each metric the agent is pushing is a custom metric. For more information on pricing for custom metrics, please see the CloudWatch pricing page.

{
        "agent": {
                "metrics_collection_interval": 60
        },
        "metrics": {
                "namespace": "ASG_Memory",
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "InstanceId": "${aws:InstanceId}"
                },
                "aggregation_dimensions" : [["AutoScalingGroupName"]],
                "metrics_collected": {
                        "mem": {
                                "measurement": [
                                         {"name": "mem_used_percent", "rename": "MemoryUtilization", "unit": "Percent"}
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}

To create a parameter

Open the AWS Systems Manager console.
In the navigation pane, choose Parameter Store, and then choose Create parameter.
In Name, enter a hierarchy and name (for example, /Test/helloWorld). For more information, see Working with parameter hierarchies.
In Description, enter a description that identifies this parameter as a test parameter.
For Parameter tier, choose Standard. For more information about advanced parameters, see Managing parameter tiers.
For Type, choose String.
Leave Data type at the default.
In Value, add the CloudWatch agent configuration JSON.
Choose Create parameter.
In the parameters list, choose the name of the parameter you just created. Verify the details on the Overview tab.

Create an IAM role to use with the CloudWatch agent

Access to AWS resources requires permissions. You can create IAM roles and users that include the permissions that you need for the CloudWatch agent to write metrics to CloudWatch and for the CloudWatch agent to communicate with Amazon EC2 and AWS Systems Manager. You use IAM roles on Amazon EC2 instances. You use IAM users with on-premises servers.

To create the IAM role for each server to run the CloudWatch agent

Sign in to the AWS Management Console and open the IAM console.
In the navigation pane, choose Roles, and then choose Create role.
Under Select type of trusted entity, choose AWS service.
Under Common use cases, choose EC2, and then choose Next: Permissions.
In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.
Choose Next: Tags.
(Optional) Add one or more tag-key value pairs to organize, track, or control access for this role, and then choose Next: Review.
For Role name, enter a name for your new role (for example, CloudWatchAgentServerRole).
(Optional) For Role description, enter a description.
Confirm that CloudWatchAgentServerPolicy appears next to Policies.
Choose Create role.

Create a launch template for an Auto Scaling group with user data

Before you can create an Auto Scaling group using a launch template, you must create a launch template that includes the parameters required to launch an EC2 instance, such as the ID of the Amazon Machine Image (AMI) and an instance type. We strongly recommend that you do not use launch configurations. They do not provide full functionality for Amazon EC2 Auto Scaling or Amazon EC2. We provide information about launch configurations for customers who have not yet migrated from launch configurations to launch templates.

To create a launch template

Open the Amazon EC2 console.
On the navigation pane, under INSTANCES, choose Launch Templates.
Choose Create launch template. Enter a name and provide a description for the initial version of the launch template.
Under Auto Scaling guidance, select the check box to have Amazon EC2 provide guidance to help create a template to use with Amazon EC2 Auto Scaling.
Under Launch template contents, fill out each required field and any optional fields to use as your instance launch specification.
Expand Advanced details to view the fields.
For IAM instance profile, you can specify an IAM instance profile to associate with the instances. When you choose an instance profile, you associate the corresponding IAM role, CloudWatchAgentServerRole, created for the EC2 instances.
For User data, you can specify user data to configure an instance during launch or to run a configuration script after the instance starts.
Choose Create launch template.

User data script

Install the CloudWatch agent package to your fleet OS and configure the agent using the Parameter Store.

#!/bin/bash
# Download CloudWatch Unified Agent for Amazon Linux 
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

# Install The Unified agent
rpm -U ./amazon-cloudwatch-agent.rpm

## Configure the agent using the SSM Parameter
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWAgent-Linux -s

Verify created metrics

Verify that the aggregated MemoryUtilization metric is the same as the average math expression of the MemoryUtilization metrics for the Amazon EC2 Auto Scaling instances.

Comparing aggregated MemoryUtilization metric on Amazon EC2 Auto Scaling group with average math expression of the Linux fleet EC2 instances individual MemoryUtilization metrics.

Figure 2: MemoryUtilization

Create an Auto Scaling group using a launch template

With launch templates, you can configure the Auto Scaling group to dynamically choose either the default version or the latest version of the launch template when a scale-out event occurs. For example, you configure your Auto Scaling group to choose the current default version of a launch template. To change the configuration of the EC2 instances to be launched by the group, create or designate a new default version of the launch template. Alternatively, you can choose the specific version of the launch template that the group uses to launch EC2 instances. You can change these selections at any time by updating the group.

To create an Auto Scaling group using a launch template

Open the Amazon EC2 console.
On the navigation bar, choose the AWS Region that you used when you created the launch template.
On the navigation pane, under AUTO SCALING, choose Auto Scaling Groups.
Choose Create an Auto Scaling group.
On the Choose launch template or configuration page, do the following:
1. For Auto Scaling group name, enter a name for your Auto Scaling group.
2. For Launch Template, choose an existing launch template.
3. For Launch template version, choose whether the Auto Scaling group uses the default, the latest, or a specific version of the launch template when scaling out.
4. Verify that your launch template supports all of the options that you are planning to use, and then choose Next.
On the Configure settings page, for Purchase options and instance types, choose Adhere to the launch template to use the EC2 instance type and purchase option that are specified in the launch template.
Under Network, for VPC, choose the VPC for the security groups that you specified in your launch template.
For Subnet, choose one or more subnets in the specified VPC. Use subnets in multiple Availability Zones for high availability. For more information about high availability with Amazon EC2 Auto Scaling, see Distributing Instances Across Availability Zones.
Choose Next. Or, you can accept the rest of the defaults, and choose Skip to review.
On the Review page, choose Create Auto Scaling group.

Create an Amazon EC2 Auto Scaling policy

We strongly recommend that you use a target tracking scaling policy to scale on a metric like average CPU utilization. Metrics that decrease when capacity increases and increase when capacity decreases can be used to proportionally scale out or in the number of instances using target tracking. This helps ensure that Amazon EC2 Auto Scaling follows the demand curve for your applications closely. For more information, see Target tracking scaling policies. You still have the option to use simple scaling policies or step scaling as an additional policy for a more advanced configuration. In most cases, step scaling policies are a better choice than simple scaling policies, even if you have only a single scaling adjustment.

To create a target tracking policy

You must use the AWS CLI to create a target tracking policy for a customized metric. Make sure the values for MetricName, Namespace, and Dimensions match your configured metric in the CloudWatch unified agent.

Create a JSON configuration file that uses a target value of 40% on average of the MemoryUtilization metric.

vim config.json

{
    "TargetValue":40.0,
    "CustomizedMetricSpecification":{
       "MetricName":"MemoryUtilization",
       "Namespace":"ASG_Memory",
       "Dimensions":[
          {
             "Name":"AutoScalingGroupName",
             "Value":"[Your ASG Name]"
          }
       ],
       "Statistic":"Average",
       "Unit":"Percent"
    }
 }

Run the put-scaling-policy CLI command using the config.json:

aws autoscaling put-scaling-policy --policy-name Memory40-target-tracking-scaling-policy --auto-scaling-group-name Memory --policy-type TargetTrackingScaling --target-tracking-configuration file://config.json

To create a step scaling policy

With step scaling, you choose scaling metrics and threshold values for the CloudWatch alarms that trigger the scaling process. You also define how your Auto Scaling group should be scaled when a threshold is breached for a specified number of evaluation periods.

To create a CloudWatch alarm

Open the CloudWatch console.
If necessary, from the navigation bar, change the AWS Region to the Region where your Auto Scaling group resides.
In the navigation pane, choose Alarms, and then choose Create alarm.
Choose Select metric.
On the All metrics tab, enter the Auto Scaling group’s name in the search field. Then, choose the MemoryUtilization metric. The Specify metric and conditions page appears, showing a graph and other information about the metric.
For Period, choose the evaluation period for the alarm (for example, 1 minute). When evaluating the alarm, each period is aggregated into one data point.
Under Conditions, do the following:
1. For Threshold type, choose Static.
2. For Whenever MemoryUtilization is, specify whether you want the value of the metric to be greater than, greater than or equal to, less than, or less than or equal to the threshold to trigger the alarm. Then, under than, enter the threshold value that you want to trigger the alarm.
Under Additional configuration, do the following:
1. For Datapoints to alarm, enter the number of data points (evaluation periods) during which the metric value must meet the threshold conditions to trigger the alarm. For example, two consecutive periods of 5 minutes would take 10 minutes to trigger the alarm.
2. For Missing data treatment, choose Treat missing data as bad (breaching threshold). For more information, see Configuring how CloudWatch alarms treat missing data.
Choose Next.
(Optional) Under Notification, you can choose or create the Amazon SNS topic you want to use to receive notifications. Otherwise, you can remove the notification now and add one later as needed.
Choose Next.
Enter a name (for example, Step-Scaling-AlarmHigh-AddCapacity) and an optional description for the alarm, and then choose Next.
Choose Create alarm.

To create step scaling policies

While you are configuring your scaling policies, you can create the alarms. Alternatively, you can use alarms that you created in the CloudWatch console, as described in the previous section.

To create a step scaling policy for scale out

Open the Amazon EC2 console.
On the navigation pane, under AUTO SCALING, choose Auto Scaling Groups.
Select the check box next to your Auto Scaling group.

A split pane opens up in the bottom part of the Auto Scaling groups page, showing information about the group that’s selected.

Verify that the minimum and maximum size limits are appropriately set. For example, if your group is already at its maximum size, you need to specify a new maximum in order to scale out. Amazon EC2 Auto Scaling does not scale your group below the minimum capacity or above the maximum capacity. To update your group, on the Details tab, change the current settings for minimum and maximum capacity.
On the Automatic scaling tab, in Scaling policies, choose Add policy.
To define a policy for scale out (increase capacity), do the following:
1. For Policy type, choose Step scaling.
2. Enter a name for the policy.
3. For CloudWatch alarm, choose your alarm. If you haven’t already created an alarm, choose Create a CloudWatch alarm and complete steps 4-13 to create an alarm that monitors CPU utilization. Set the alarm threshold to greater than or equal to 80 percent.
4. Use Take the action to specify the change in the current group size that this policy will make when executed. You can add a specific number of instances or a percentage of the existing group size or set the group to an exact size.

For example, choose Add, enter 30 in the next field, and then choose percent of group. By default, the lower bound for this step adjustment is the alarm threshold and the upper bound is positive infinity.

1. To add another step, choose Add step and then define the amount by which to scale and the lower and upper bounds of the step relative to the alarm threshold.
2. To set a minimum number of instances to scale, update the number field in Add capacity units in increments of at least 1 capacity units.
3. Specify an instance warm-up value for Instances need, which allows you to control the amount of time until a newly launched instance can contribute to the CloudWatch metrics.
Choose Create.

To create a step scaling policy for scale in

Choose Add policy to continue where you left off after you created a policy for scale out.
To define a policy for scale in (decrease capacity), do the following:
1. For Policy type, choose Step scaling.
2. Enter a name for the policy.
3. For CloudWatch alarm, choose your alarm. If you haven’t already created an alarm, choose Create a CloudWatch alarm and complete steps 4-13 to create an alarm that monitors CPU utilization. Set the alarm threshold to less than or equal to 40 percent.
4. Use Take the action to specify the change in the current group size that this policy will make when executed. You can remove a specific number of instances or a percentage of the existing group size or set the group to an exact size.

For example, choose Remove, enter 2 in the next field, and then choose capacity units. By default, the upper bound for this step adjustment is the alarm threshold and the lower bound is negative infinity.

1. To add another step, choose Add step and then define the amount by which to scale and the lower and upper bounds of the step relative to the alarm threshold.

Choose Create.

Cleanup

To avoid ongoing charges to your account, delete the resources you created in this walkthrough.

If you used the AWS CloudFormation template to create resources, then delete the stack.

Conclusion

In this blog post, I showed you how to create an Amazon EC2 Auto Scaling policy on a custom CloudWatch metric, such as memory usage percentage. I also showed you how to configure the CloudWatch agent to send memory utilization metric for Linux OS. For more information, see CloudWatch Agent Configuration File in the Amazon CloudWatch User Guide.

AWS Cloud Operations Blog