EC2 Container Service (ECS) Update – Access Private Docker Repos & Mount Volumes in Containers

Amazon EC2 Container Service  (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run distributed applications on a managed cluster of Amazon EC2 instances.

My colleague Chris Barclay sent a guest post to spread the word about two additions to the service. As Chris explains below, you can use images stored in private Docker repositories. You can also store and share information between containers using data volumes from the host. Let’s see what Chris has to say!

— Jeff;

Use Images Stored in Private Docker Repositories
The Amazon ECS agent can now authenticate with Docker registries, including Docker Hub. Registry authentication lets you use Docker images from private repositories in your Task Definitions. Here’s how to set it up:

1. Create a private S3 object named ecs.config in an S3 bucket with your repository’s credentials (you can get the credentials from your .dockercfg file):

   ECS_ENGINE_AUTH_TYPE=dockercfg
   ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"YOUR_AUTH_CODE","email":"email@example.com"}}
   
   

2. Add a policy to the IAM role used by the Container Instances in your ECS cluster to provide access to the object created in step 1:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "s3:GetObject"
         ],
       "Sid": "Stmt0123456789",
       "Resource": [
         "arn:aws:s3:::YOUR_BUCKET/YOUR_OBJECT"
       ],
       "Effect": "Allow"
       }
     ]
   }
   

3. When launching an EC2 instance, in the Advanced Details drop-down paste the following script into the User data text box:

   #!/bin/bash
   yum install -y aws-cli
   aws s3 cp s3://YOUR_BUCKET/YOUR_OBJECT /etc/ecs/ecs.config
   

The container instances launched in step 3 can now pull private images referenced in an ECS Task Definition.

Mount Volumes in Containers
ECS Task Definitions now provide a way for containers to store and share information using data volumes. For data that should persist between tasks, such as a download cache, you can reference a location on the host as shown in the following volume definition:

"volumes": [
   {
    "name": "cache",
    "host": {
      "sourcePath": "/var/lib/MyApp/cache/"
    }
  } ]

You can then reference the volume by name and specify the path to mount the volume in the container definition:

"containerDefinitions":[
  {
    "name": "webserver",
     ...
    "mountPoints": [
    {
      "sourceVolume": "cache",
      "containerPath": "/usr/src/app/cache"
    } ]
    ...

If you don’t need your data to persist between task runs then you can specify an “empty” volume. By letting Docker manage the host storage, you don’t need to worry about creating or deleting the volume on the host; Docker creates a volume that persists for the lifetime of the task. Here is a Task Definition snippet that creates a volume that is managed by Docker:

"volumes": [
    {
      "name": "logs",
      "host": {}
    }

Docker also supports the ability to share volumes between containers. For example, you may take the logs your Apache server writes to /var/log/www and push them to a central repository using a cron job running in the Apache container. Another option is to create a backup job container with a backup daemon that references the shared logs volume using the volumesFrom attribute. Now the backup daemon can tar the logs in the shared volume periodically and store them in Amazon Simple Storage Service (Amazon S3). Here’s how you would reference a shared volume:

"containerDefinitions":[
  {
    "name": "backupManager",
     ...    
    "volumesFrom": [
        {
            "sourceContainer": "Apache",
            "readOnly": true
        }
      ]
    }

Available Now
These features are available now and you can start using them today (to borrow Jeff’s phrase). For more information, read the documentation on private Docker repositories and mounting volumes in containers.

— Chris Barclay, Principal Product Manager