Automating ECR Authentication on Marathon with the Amazon ECR Credential Helper

This is a guest post from Erin McGill and Brandon Chavis, Partner Solution Architects with AWS.

Are you running the Datacenter Operating System (DC/OS) on AWS and want to leverage the Amazon EC2 Container Registry (Amazon ECR) without managing Docker registry credentials or scheduling a periodic job to authenticate with ECR on your DC/OS hosts?

In this post, we’ll show you how to use Marathon, a native, production-grade container orchestrator for DC/OS, to automate authentication with ECR. This method uses the ECR Credential Helper to pull and run Docker images seamlessly, without scheduled re-authentication tasks or storing Docker credentials on the Marathon agents.

To learn more about DC/OS on AWS, check out our previous blog post.

Authenticating to EC2 Container Registry

To pull an image from an ECR hosted private repository, you must first obtain a valid login token for Docker to use. Because Docker doesn’t use IAM directly, you can first call the aws ecr get-login command from the AWS Command Line Interface (AWS CLI) to request a temporary login token. This command returns a docker login command that you can use to authenticate with ECR:

docker login -u AWS -p temp-password -e none 
https://aws_account_id.dkr.ecr.region.amazonaws.com

This temporary token lasts for 12 hours. When the token expires, you’ll need to request a new one. We can streamline this process and remove the need to either manually re-authenticate or write a program to call aws ecr get-login by using the Amazon ECR Docker Credential Helper.

The ECR Credential Helper is a tool that makes it easier to use Amazon ECR based on Docker credential helpers. Docker credential helpers is a suite of programs that allow you to use external credential stores for your Docker credentials.

When you use the ECR Credential Helper, you no longer need to schedule a job to get temporary tokens and store those secrets on the hosts, and the ECR Credential Helper can get IAM permissions from your AWS credentials, such as an IAM EC2 Role, so there are no stored authentication credentials in the Docker configuration file. The configuration file tells Docker to use the credential helper, and the helper gets an ECR authorization token that is used by Docker for each call to ECR.

Creating a Docker container to run the ECR Helper

On AWS, DC/OS runs on CoreOS, a lightweight host system, and uses Docker containers for all applications, so nothing is installed on the host. To adhere to the CoreOS model, we developed a solution to use a Docker container that compiles the ECR credential helper binary and puts the binary file and a compressed TAR credential file on the host.

After running the container, the agents will be able to automate authentication with ECR and pull containers from the private repositories.

To use this solution, create an empty directory called aws-ecr-helper. Within that directory, create a folder named .docker. Create a Docker configuration file called config.json and save it in the new, empty .docker folder. The config.json file consists of a single line:

{ "credsStore": "ecr-login" }

Following the documentation on how to use a private Docker registry with Marathon, create a compressed TAR file that includes the .docker folder and its contents:

tar -czf docker.tar.gz .docker

A Dockerfile is a file that contains all the commands to create a Docker image. Using a Dockerfile, you can create an image to:

Get a zipped archive of the ECR Credential Helper repository.
Unzip the repository archive.
Create the ECR Credential Helper binary.
Place the binary on the host system.
Place the docker.tar.gz file on the host system.

The Dockerfile we used looks like this:

# Because the Amazon ECR Helper is a go binary, we are using 
# the official golang docker image as the base image
FROM golang:1.6

RUN apt-get update && apt-get install -y \
    unzip \
    && rm -rf /var/lib/apt/lists/*

# Marathon requires a gzipped credntial file - this compressed tarball contains 
# .docker/config.json

# The JSON file contains the following single line: { "credsStore": "ecr-login" }
# There is no secret or confidential account information needed

COPY docker.tar.gz /tmp/

# Creating the necessary github directories and pulling a zip of the master branch 
# from the repository using the wget command to avoid installing the Git client

RUN mkdir -p src/github.com/awslabs/amazon-ecr-credential-helper/ && \
    wget -O src/github.com/awslabs/amazon-ecr-credential-helper/master.zip \
    --no-check-certificate https://github.com/awslabs/amazon-ecr-credential-helper/archive/master.zip

# Setting the new working directory, expanding the ECR Helper code, and cleanup

WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper/
RUN unzip master.zip && \
    mv amazon-ecr-credential-helper-master/* . && \
    rm -rf amazon-ecr-credential-helper-master && \
    rm -f master.zip

# Compile the binary with make - the binary will be created in the #/go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local
# directory inside the container
#
# To ensure that the host has the necessary files after the container runs and is removed, the user has to mount 2 volumes from the host
#  - mapped to the container's /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local directory
#  - mapped to the container's /data directory

CMD /usr/bin/make && cp /tmp/docker.tar.gz /data

Save the Dockerfile in the same directory as the docker.tar.gz file.

The aws-ecr-helper directory now contains:

The .docker folder, which contains the config.json file
The docker.tar.gz file created from the .docker folder
the Dockerfile from the example above

Build the container using the command:

docker build –t marathon-ecr-helper .

Note: If you previously built this Docker image on the same host, run the docker build command with the --no-cache option to ensure that the container pulls the latest master branch of the ECR helper.

Running the Amazon ECR Credential Helper

To test that our Docker image compiles the binary successfully, we can use the docker run command on your build host:

docker run -it --rm -v /opt/mesosphere/bin:/go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/ -v /etc:/data marathon-ecr-helper

This command compiles the ECR Credential Helper and places the resulting ECR Credential Helper binary bin and compressed TAR credential file on the host. The -v flag bind-mounts a host directory into the container. In this case, there are two mount points:

-v /opt/mesosphere/bin:/go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/

and:

-v /etc:/data

The first mount from the host has to be a directory in the PATH environment variable of the Marathon process owner. In our example, we used /opt/mesosphere/bin. In the DC/OS documentation for using a private Docker registry, the example location for the compressed credential file is /etc, so we used this location as well.

Run the container with the -it --rm flags to view what the container is doing and to remove the container after its job is finished. The example command outputs the following to the screen:

. ./scripts/shared_env && ./scripts/build_binary.sh  ./bin/local
Built ecr-login

You can see what the container is executing, any errors that occurred, and a notification that the build is complete and successful.

The container is now ready to be tagged and sent to the repository. Tag the image by using the tag command:

docker tag marathon-ecr-helper public-repo/marathon-ecr-helper:latest

Then push the container to the registry:

docker push public-repo/marathon-ecr-helper:latest

You should store the Docker image in a public repository so Marathon doesn’t need to authenticate it in order to pull the ECR Credential Helper image. When the image is in the repository, you can create an application within Marathon to pull the image and run the container to place the helper binary and necessary configuration on the Marathon agent nodes.

If you want to use the ECR Credential Helper on your development machine, ensure that the config.json file is present and that the binary is in a directory that is in the environment PATH variable.

Preparing the DC/OS stack with CloudFormation

If you are not already running DC/OS or want to launch a new DC/OS test environment, first, download the CloudFormation template. We will use it to launch the DC/OS cluster in this example. If you are already running DC/OS launched from a CloudFormation template, you’ll need to update your stack with these changes to use the automated solution presented in this blog post.

To access ECR with DC/OS on AWS, you need to make sure that your Marathon agent nodes can access the ECR service and that the CoreOS version can support Docker credential helpers.

The IAM instance profiles for the EC2 instances need to contain read-only permissions for ECR, so we’ve modified the CFN template by adding these ECR permissions to the EC2 IAM Roles:

"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage"

To use the compiled ECR Credential Helper, we also need to modify the version of CoreOS in the Cloudformation template. Docker credential helper support was introduced in Docker version 1.11. As of this writing, Docker version 1.11 is available in the Beta CoreOS release.

You can choose the tab for the Beta channel on the CoreOS EC2 page to find the AMI ID for the region where you want to launch DC/OS. You will replace the existing AMI IDs with the new Beta Channel AMI ID in RegionToAmi of the Mappings section in the CloudFormation template.

In our example, we select 2 public agents and 2 private agents to run in our DC/OS cluster.

Creating the ECR Credential Helper application in Marathon

Once the stack has the correct permissions and is running with the correct version of CoreOS, you can log in to the DC/OS stack and create a Marathon application for the ECR Credential Helper containers.

The Marathon application consists of the following code:

  1 {
  2   "id": "/aws-ecr-helper",
  3   "cmd": null,
  4   "cpus": 1,
  5   "mem": 256,
  6   "disk": 0,
  7   "instances": 0,
  8   "acceptedResourceRoles": [
  9         "*",
 10         "slave_public"
 11   ],
 12   "container": {
 13     "type": "DOCKER",
 14     "volumes": [
 15       {
 16         "containerPath": "/data",
 17         "hostPath": "/etc",
 18         "mode": "RW"
 19       },
 20       {
 21         "containerPath": "/go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/",
 22         "hostPath": "/opt/mesosphere/bin/",
 23         "mode": "RW"
 24       }
 25     ],
 26     "docker": {
 27       "image": "public-repo/marathon-ecr-helper:latest",
 28       "network": "HOST",
 29       "privileged": false,
 30       "parameters": [],
 31       "forcePullImage": true
 32     }
 33   },
 34   "portDefinitions": [
 35     {
 36       "port": 10000,
 37       "protocol": "tcp",
 38       "labels": {}
 39     }
 40   ]
 41 }

Let’s break down the configuration and identify the important sections of code.

Line 2 identifies the name you give the application in Marathon.

Line 7 tells Marathon to launch 0 Docker instances for this application.

In lines 8-10, you can ensure that when you deploy your test web container, the ECR Credential Helper container will have been deployed to it. The resource role is an asterisk (*) and “slave_public” so the Docker container for the credential helper will be deployed to Marathon workers that are available inside and outside the environment.

Lines 14-18 and 19-23 show the two mount points we will be using when running this container. The containerPath is the path within the Docker container, the hostPath is the directory path on the agent node.

The first entry mounts /etc from the host into the container at the /data directory. After the Docker container runs, the docker.tar.gz file is copied to the /data location. Once the container finishes running its command, the TAR file will be in /etc on the host.

The second entry mounts /opt/mesosphere/bin/ from the host into the container at the /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/ location. When the container runs, it compiles the Go code into a binary. When the container has completed its job, the binary will be left on the host at /opt/mesosphere/bin/ so Marathon can use it to authenticate users when pulling images from ECR.

Lines 26-32 define the repository and the image to launch as well as any parameters or specifications for the running container.

Now that you’ve created the Marathon application for the ECR Credential Helper, you can scale up from 0 instances (line 7 in the above JSON document) to have Marathon launch the containers.

In our example, we launched the DC/OS stack with the private agent node count set to 2 and the public agent node count set to 2, so we should scale the application up to 4: one for each agent node launched. The container spins up, places the compiled binary and compressed TAR file, and then stops.

Once the container has been run on all your agents, you can scale the ECR Credential Helper application back down to 0. There is no need to run the application again until you need to replace an agent or scale up your DC/OS cluster.

Deploying a ‘Hello World!’ container from ECS

To test that you can pull from a private repository, you can create a simple container based on the official Nginx container. If you do not already have an ECR repository to push to, either create one in the console or use the AWS CLI command aws ecr create-repository. Save the URI for the created repository; you will use it when tagging and pushing the sample container image.

$ aws ecr create-repository --repository-name marathon-nginx-example
{
    "repository": {
        "registryId": " aws_account_id ",
        "repositoryName": "marathon-nginx-example",
        "repositoryArn": "arn:aws:ecr:region:aws_account_id:repository/marathon-nginx-example",
        "repositoryUri": " aws_account_id.dkr.ecr.region.amazonaws.com/marathon-nginx-example"
    }
}

Create an index.html page for the new container:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to Nginx - in a Docker container!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Hello World!</h1>
<p>This is an image pulled from ECR</p>

</body>
</html>

The Dockerfile to place the new index.html page inside the container:

# This is for an nginx test
FROM nginx:latest

RUN apt-get -y update
COPY index.html /usr/share/nginx/html/index.html

EXPOSE 80

To build the Docker image, use the command:

docker build -t marathon-nginx-example .

Next, if you have the ECR Credential Helper and proper configuration on your development machine, you can push the image to an ECR repository called marathon-nginx-example. Tag the image and upload it to your private ECR repository:

docker tag marathon-nginx-example aws_account_id.dkr.ecr.region.amazonaws.com /marathon-nginx-example:latest

docker push aws_account_id.dkr.ecr.region.amazonaws.com/marathon-nginx-example:latest

Your modified Nginx container is now in ECR. You will configure Marathon to pull the new image from the private repository and run the web server. To do this, you’ll need to create an application configuration for the new Nginx container. It needs to expose port 80 on the agent, so you can view the modified index page, and it needs to use the compressed configuration file that was placed on the host by the Docker container for ECR Credential Helper, so Marathon knows to use the ECR Credential Helper binary.

Here’s the application definition that will pull the image and run the newly created Nginx container:

{
  "id": "/marathon-nginx-example",
  "cmd": null,
  "cpus": 1,
  "mem": 256,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": [
    "slave_public"
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": " aws_account_id.dkr.ecr.region.amazonaws.com/marathon-nginx-example",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "servicePort": 0,
          "protocol": "tcp",
          "name": "nginx",
          "labels": {
            "VIP_0": "80"
          }
        }
      ],
      "privileged": true,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "portDefinitions": [
    {
      "port": 10003,
      "protocol": "tcp",
      "labels": {}
    }
  ],
  "uris": [
    "file:///etc/docker.tar.gz"
  ],
  "fetch": [
    {
      "uri": "file:///etc/docker.tar.gz",
      "extract": true,
      "executable": false,
      "cache": false
    }
  ]
}

This example configuration pulls the new image that you committed to the ECR; specifies the public agents so that when you scale your application up, it deploys to publicly available EC2 instances; bridges port 80 on the host to port 80 on the container instance; and uses the URI to fetch the compressed configuration file from where the ECR Credential Helper placed it.

You can now scale up the application and wait for it to be launched on the public agents. To view the new page, get the DNS host name for the public agent ELB load balancer that was created when you launched the DC/OS stack. You can find it in the Outputs section of your CloudFormation stack.

There it is! You just deployed a Docker container from a private repository without having to store and manage access and secret keys, user names and passwords, or create a scheduled job on each host.

To recap, we created a Docker image that compiled the ECR Docker Credential Helper and places the compiled binary and compressed configuration tar file on a DC/OS host. We then pushed this container to a public repository. Next, we modified the DC/OS CloudFormation template to include a Beta version of the CoreOS AMI that includes Docker 1.11, which allows us to use Docker Credential helpers and added IAM policies to allow the DC/OS agents to perform specific actions in ECR.

We then launched the modified CloudFormation template, created an application in Marathon to pull the credential-helper image from the public repository, and scheduled the container on the DC/OS agents.

Finally, to test that the compiled binary is in place and works as expected, we created a sample Nginx Docker image with a modified index.html that we then pushed to a private ECR repository and launched on the DC/OS agents.

To learn more about ECR, visit https://aws.amazon.com/ecr/

To learn more about DC/OS, visit https://dcos.io/

AWS Partner Network (APN) Blog