EC2 Container Service Update – Container Registry, ECS CLI, AZ-Aware Scheduling, and More

I’m really excited by the Docker-driven, container-based deployment model that is quickly becoming the preferred way to build, run, scale, and quickly update new applications. Since we launched Amazon EC2 Container Service last year, we have seen customers use it to host and run their microservices, web applications, and batch jobs.

Many developers have told me that containers have allowed them to speed up their development, testing, and deployment efforts. They love the fact that individual containers can hold “standardized” application components, each of which can be built using the language, framework, and middleware best suited to the task at hand. The isolation provided by the containers gives them the freedom to innovate at a more granular level instead of putting the entire system at risk due to large-scale changes.

Based on the feedback that we have received from our contingent of Amazon ECS and Docker users, we are announcing some powerful new features  – the Amazon EC2 Container Registry and the Amazon EC2 Container Service CLI. We are also making the Amazon ECS scheduler more aware of Availability Zones and adding some new container configuration options.

Let’s dive in!

Amazon EC2 Container Registry
Docker (and hence EC2 Container Service) is built around the concept of an image. When you launch a container, you reference an image, which is pulled from a Docker registry. This registry is a critical part of the deployment process. Based on conversations with our customers, we have learned that they need a registry that is highly available and exceptionally scalable, and globally accessible, with the ability to support deployments that span two or more AWS regions. They also want it to integrate with AWS Identity and Access Management (IAM) to simplify authorization and to provide fine-grained control.

While customers could meet most of these requirements by hosting their own registry, they have told us that this would impose an operational burden that they would strongly prefer to avoid.

Later this year we will make the Amazon EC2 Container Registry (Amazon ECR) available. This fully managed registry will address the issues that I mentioned above by making it easy for you to store, manage, distribute, and collaborate around Docker container images.

Amazon ECR is integrated with ECS and will be easy to integrate into your production workflow. You can use the Docker CLI running on your development machine to push images to Amazon ECR, where Amazon ECS can retrieve them for use in production deployments.

Images are stored durably in S3 and are encrypted at rest and in transit, with access controls via IAM users and roles. You will pay only for the data that you store and for data that you transfer to the Internet.

Here’s a sneak peek at the console interface:

You can visit the signup page to learn more and to sign up for early access. If you are interested in participating in this program, I would encourage you to sign up today.

We are already working with multiple launch partners including Shippable, CloudBees, CodeShip, and Wercker to provide integration with Amazon ECS and Amazon ECR, with a focus on automatically building and deploying Docker images. To learn more, visit our Container Partners page.

Amazon EC2 Container Service CLI
The ECS Command Line Interface (ECS CLI) is a command line interface for Amazon EC2 Container Service (ECS) that provides high level commands that simplify creating, updating and monitoring clusters and tasks from a local development environment.

The Amazon ECS CLI supports Docker Compose, a popular open-source tool for defining and running multi-container applications. You can use the ECS CLI as part of your everyday development and testing cycle as an alternative to the AWS Management Console.

You can get started with the ECS CLI in a couple of minutes. Download it (read the directions first), install it, and then configure it as follows (you have other choices and options, of course):

$ ecs-cli configure --region us-east-1 --cluster my-cluster

Launch your first cluster like this:

$ ecs-cli up --keypair my-keys --capability-iam --size 2

Docker Compose requires a configuration file. Here’s a simple one to get started (put this in docker-compose.yml):

web:
  image: amazon/amazon-ecs-sample
  ports:
  - "80:80"

Now run this on the cluster:

$ ecs-cli compose up
INFO[0000] Found task definition TaskDefinition=arn:aws:ecs:us-east-1:980116778723:task-definition/ecscompose-bin:1
INFO[0000] Started task with id:arn:aws:ecs:us-east-1:9801167:task/fd8d5a69-87c5-46a4-80b6-51918092e600

Then take a peek at the running tasks:

$ ecs-cli compose ps
Name                                      State    Ports
fd8d5a69-87c5-46a4-80b6-51918092e600/web  RUNNING  54.209.244.64:80->80/tcp

Point your web browser at that address to see the sample app running in the cluster.

The ECS CLI includes lots of other options (run it with –help to see all of them). For example, you can create and manage long-running services. Here’s the list of options:

The ECS CLI is available under an Apache 2.0 license (code is available at https://github.com/aws/amazon-ecs-cli) and we are looking forward to seeing your pull requests.

New Docker Container Configuration Options
A task definition is a description of an application that lets you define the containers that are scheduled together on an EC2 instance. Some of the parameters you can specify in a task definition include which Docker images to use, how much CPU and memory to use with each container, and what (if any) ports from the container are mapped to the host container.

Task definitions now support lots of Docker options including Docker labels, working directory, networking disabled, privileged execution, read-only root filesystem, DNS servers, DNS search domains, ulimits, log configuration, extra hosts (hosts to add to /etc/hosts), and security options for Multi-Level Security (MLS) systems such as SELinux.

The Task Definition Editor in the ECS Console has been updated and now accepts the new configuration options:

For more information, read about Task Definition Parameters.

Scheduling is Now Aware of Availability Zones
We introduced the Amazon ECS service scheduler earlier this year as a way to easily schedule containers for long running stateless services and applications. The service scheduler optionally integrates with Elastic Load Balancing. It ensures that the specified number of tasks are constantly running and restarts tasks if they fail. The service scheduler is the primary way customers deploy and run production services with ECS and we want to continue to make it easier to do so.

Today the service scheduler is availability zone aware. As new tasks are launched, the service scheduler will spread the tasks to maintain balance across AWS availability zones.

Amazon ECS at re:Invent
If you are at AWS re:Invent and want to learn more about how your colleagues (not to mention your competitors) are using container-based computing and Amazon ECS, check out the following sessions:

• CMP302 – Amazon EC2 Container Service: Distributed Applications at Scale (to be live streamed).
• CMP406 – Amazon ECS at Coursera.
• DVO305 – Turbocharge Your Continuous Deployment Pipeline with Containers.
• DVO308 – Docker & ECS in Production: How We Migrated Our Infrastructure from Heroku to AWS.
• DVO313 – Building Next-Generation Applications with Amazon ECS.
• DVO317 – From Local Docker Development to Production Deployments.

— Jeff;