Scaling Amazon ECS Services Automatically Using Amazon CloudWatch and AWS Lambda

AWS Partner Classmethod contributed this great blog that shows how to use Amazon CloudWatch and AWS Lambda to create a scaling function for Amazon EC2 Container Service (Amazon ECS).

Amazon ECS is a highly scalable, high-performance, container management service that supports Docker containers and allows you to run applications easily on a managed cluster of Amazon EC2 instances. Amazon ECS also includes a service scheduler to manage long-running applications. The service scheduler maintains application availability and allows you to scale your containers up or down to meet your application’s capacity requirements. The service scheduler allows you to distribute traffic across your containers using Elastic Load Balancing. Amazon ECS automatically registers and deregisters your containers from the associated load balancer. The service scheduler also automatically attempts to recover containers that become unhealthy or stop running, to ensure you have the desired number of healthy containers supporting your application. You can use the service scheduler to scale up and down the number of containers supporting your application, and you can update your application by changing its definition or using a new image. For more information about Amazon ECS services, see this blog.

In this post, we show how to use Amazon CloudWatch custom metrics and AWS Lambda to create a function that automatically increases and decreases the capacity of an Amazon ECS service.

You will create the following resources in this post:

• Amazon ECS cluster and service: Amazon EC2 compute instances for Docker containers and an Amazon ECS service to run your application
• Amazon CloudWatch alarm: Identify when a CloudWatch metric goes above/below a threshold to increase/decrease the tasks (containers) running in the Amazon ECS service
• Amazon SNS topic: Send a notification when the Amazon CloudWatch alarm is triggered
• AWS Lambda function: Increase/decrease the desiredCount of the Amazon ECS service in response to the CloudWatch alarm

A benefit of this architecture is there are no fixed processes that you have to run to monitor the CloudWatch metrics, which can help reduce cost and operational overhead.

Step 1: Build an ECS cluster and service, using the instructions in the Getting Started with Amazon ECS topic.

Step 2: Add a CloudWatch alarm on Elastic Load Balancing metrics and configure an Amazon SNS topic.

Identify the load balancer that is being used with the ECS service scheduler, then navigate to the CloudWatch console and find the load balancer’s metrics. Choose Create Alarm. In this example, we use RequestCount as a metric; you can use other metrics, such as Latency.

In the window, input values for Name and Description. Then, set the threshold for RequestCount as the number of requests per minute that should be scaled out (100, in this case).

On the Actions page, add a new SNS topic. Choose New List and input a value for Name (such as ECSServiceELBAlarm) and an email address for notification, and then choose Create Alarm.

When you see a message “Confirm new email addresses”, you should receive a confirmation mail. Choose the Confirm subscription link in the body of the email.

Step 3: Use IAM to configure permissions for a Lambda function.

In the IAM console, create an IAM role to execute a Lambda function. Use “lambda_exec_role” as the value for Name and then choose AWS Service Roles, AWS Lambda with an empty Attach Policy value. Choose the link under Inline Policies, choose Custom Policy, and then enter a Policy Name value and the following Policy Document text. This policy allows the Lambda function to update the ECS service:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecs:DescribeServices",
        "ecs:UpdateService"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Step 4: Create the Lambda function and register the SNS topic.

Now, define a Lambda function. From the Lambda console, choose Create a Lambda function and enter the following:

• Name: Any name (e.g., scaleOutServiceCount).
• Description: Any description
• Code entry type: Edit code inline
• Code template: None (copy and paste from below)
• Handler name: handler (default value)
• Role: The IAM role that was just created (lambda_exec_role)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

console.log('Loading event');
var aws = require('aws-sdk');
 
exports.handler = function(event, context) {
  var ecsService = 'sample-webapp';
  var ecsRegion = 'us-west-2';
  var maxCount = 2;
 
  var ecs = new aws.ECS({region: ecsRegion});
  ecs.describeServices({services:[ecsService]}, function(err, data) {
    if (err) {
      console.log(err, err.stack);
    } else {
      var desiredCount = data.services[0].desiredCount;
      if (desiredCount < maxCount) {
        desiredCount++;
        var params = {
          service:      ecsService,
          desiredCount: desiredCount
        };
        ecs.updateService(params, function(err, data) {
          if (err) {
            console.log(err, err.stack);
          } else {
            console.log(data);
            context.succeed();
          }
        });
      } else {
        console.log('Service count is already max.');
        context.fail();
      }
    }
  });
};

Step 5: Customize the Lambda function

The code is very simple, but you will need to customize it for your service and region. Here are some notes:

• Lines 5 and 6: Change to match the service name and region you are using.
• Line 7: Define maxCount to configure the maximum number of tasks for scaling out. If you use RequestCount, the alarm can be triggered repeatedly.
• Line 14: Gets the current task count from the ECS service.
• Line 16: Increment DesiredCount by 1 to scale as long as the count is less than the max count defined in line 7.
• Line 21: Update the service scheduler configuration.

After creating the function, choose scaleOutServiceCount from the list of functions, and choose Actions, Add event source.

Enter SNS for Event Source Type, ECSServiceELBAlarm for SNS topic, and then choose Submit.

Step 6: Try it out

You can trigger an alarm by adding load to the load balancer, but testing the Lambda function is much easier. Choose the function, choose Actions, Test/Edit, and then Invoke. There are execution logs in the Execution logs pane at bottom of the window and you can see that the service’s desiredCount is increased to 2.

Also, you can see that the service is scaled out in the ECS console.

Conclusion

This post shows how to create a simple function using Lambda to scale ECS services automatically. The combination of AWS Lambda and Amazon ECS can automate many tasks. We look forward to your feedback.

Original post: http://dev.classmethod.jp/cloud/ecs-autoscale-by-lambda/
Author: Ryuta Otaki, Senior Solutions Architect, Classmethod
Translated by: Ryosuke Iwanaga, AWS Solution Architect in Japan