AWS Compute Blog
Running an Amazon ECS Task on Every Instance
My colleague Matt McClean sent a nice guest post that demonstrates how to start a task on every ECS instance in a cluster.
—–
Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.
When deploying applications in a container-based architecture, there is often a need to run a specific agent on every Amazon EC2 instance to deal with operations or security concerns such as monitoring, security, metrics, service discovery, logging, etc. The traditional method is to have the EC2 instance call the Docker run command for each agent in the user data script or in some init system such as Upstart or Systemd. While this works, it has some disadvantages because Amazon ECS has no knowledge of the container and thus cannot track the CPU, memory, or port used. It can also be difficult to manage when new agents need to be added, changed, or removed from the instances, because changes need to be made locally to each EC2 instance and require you to connect using SSH into each machine to start and stop the containers.
This blog post shows you how to launch a specific task definition upon ECS instance bootup for a particular cluster with a user data script.
Example setup
In this example, you build an example application and run the cAdvisor (container advisor) agent on every ECS instance. This provides you with an understanding of resource usage and performance characteristics of your running containers. For more information about cAdvisor, go to the GitHub cadvisor repository.
To start, make sure the following requirements are in place:
- You have an AWS account. For more information, see the documentation.
- A VPC exists for the account with DNS support enabled and at least one public subnet. For more information, see the documentation.
- An EC2 key pair has been generated and you have access to the private key file (*.pem). For more information, see the documentation.
To create the Amazon ECS cluster
- In the ECS console, choose Create cluster.
- Give the cluster a unique name and choose Create.
To create a task definition for the cAdvisor metrics agent
- In the ECS console, choose Task Definitions on the left menu.
- On the Task Definitions page, choose Create new task definition.
- On the JSON tab, paste the JSON text below into the text area:
{
"family": "cadvisor",
"containerDefinitions": [
{
"name": "cadvisor",
"image": "google/cadvisor",
"cpu": 10,
"memory": 300,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8080
}
],
"essential": true,
"mountPoints": [
{
"sourceVolume": "root",
"containerPath": "/rootfs",
"readOnly": true
},
{
"sourceVolume": "var_run",
"containerPath": "/var/run",
"readOnly": false
},
{
"sourceVolume": "sys",
"containerPath": "/sys",
"readOnly": true
},
{
"sourceVolume": "var_lib_docker",
"containerPath": "/var/lib/docker",
"readOnly": true
}
]
}
],
"volumes": [
{
"name": "root",
"host": {
"sourcePath": "/"
}
},
{
"name": "var_run",
"host": {
"sourcePath": "/var/run"
}
},
{
"name": "sys",
"host": {
"sourcePath": "/sys"
}
},
{
"name": "var_lib_docker",
"host": {
"sourcePath": "/var/lib/docker/"
}
}
]
}
4. On the Builder tab, verify your information and choose Create.
To create an IAM ECS role
The IAM role has the standard ECS IAM permissions, and also allows the instance to call the StartTask operation to invoke the task definition on the same instance.
- Create a new ECS IAM role based on the policy document described below. For more information, see Amazon ECS Container Instance IAM Role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:CreateCluster",
"ecs:RegisterContainerInstance",
"ecs:DeregisterContainerInstance",
"ecs:DiscoverPollEndpoint",
"ecs:Submit*",
"ecs:Poll",
"ecs:StartTask"
],
"Resource": [
"*"
]
}
]
}
To launch an ECS instance with defined tasks
The user data code provided configures the name of the ECS cluster that the EC2 instance will join, retrieves the ECS container instance ARN from the ECS container agent introspection API, and invokes the ECS StartTask operation call to start the task definition named cadvisor:1 that was created earlier.
1. Launch an ECS instance. For more information, see Launching an Amazon ECS Container Instance, except use the following code in the User data field. Remember to replace the value for your_cluster_name.
#!/bin/bash
cluster="your_cluster_name"
echo ECS_CLUSTER=$cluster >> /etc/ecs/ecs.config
start ecs
yum install -y aws-cli jq
instance_arn=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $NF}' )
az=$(curl -s http://instance-data/latest/meta-data/placement/availability-zone)
region=${az:0:${#az} - 1}
aws ecs start-task --cluster $cluster --task-definition cadvisor:1 --container-instances $instance_arn --region $region
Note: This should also be added to any init script (such as upstart or systemd) so that the rebooted instance starts the task.
To verify that the tasks were launched
- In the AWS Management Console, choose EC2 Container Service in the region where you launched the instance.
- On the Clusters tab, choose the ECS cluster previously configured.
- On the Tasks tab, you should see the task definition named cadvisor:1 in the RUNNING state.
Conclusion
If you have followed all of these steps, you should now have a system where every ECS instance that is started in the cluster gets the task definition called cadvisor launched on the instance. You can use a similar approach to launch the tasks you want on every instance in your cluster by changing the task definition in the user data example.