Containers
Building container images on Amazon ECS on AWS Fargate
Building container images is the process of packaging an application’s code, libraries, and dependencies into reusable file systems. Developers create a Dockerfile alongside their code that contains all the commands to assemble a container image. This Dockerfile is then used to produce a container image using a container image builder tool, such as the one built into Docker Engine. The resulting container image is used to create containers in containerized environments such as Amazon ECS and EKS. As part of the development workflow, a developer builds container images locally on their machine, for example, running a docker build command against a local Docker Engine.
While this practice works well when there’s only one developer who’s writing the code and building it, it’s not a scalable process. Given that multiple developers simultaneously modify code in a typical development team, one developer cannot be responsible for building container images. DevOps engineers solve this problem using continuous delivery (CD) pipelines where developers check-in their code in a central code repository such as a Git repository, and container builds are automated using tools like Jenkins or CodePipeline. Through customer feedback, we have learned that many DevOps teams that manage their CD pipelines choose to run it on Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Container orchestrators like ECS and EKS simplify scaling the infrastructure based on the demands on the CD system. For example, in Jenkins, ECS can autoscale EC2 instances as Jenkins pipelines get triggered and additional compute capacity to run the builds is required. Customers have also expressed interest in running their CD workloads on Fargate as it eliminates the need to manage servers.
Building container images in containers
In contrast with building containers on your local machine, Jenkins (or a similar tool) running in an ECS cluster will build container images inside a running container. However, common container image builders, such as the one included in the Docker Engine, cannot run in the security boundaries of a running container. Restricted access to Linux Systems Calls (via seccomp) and Linux Security Modules (AppArmour or SELinux) prevent Docker Engine from running inside a container.
Therefore, customers have two options if they want to build containers images using the traditional docker build
method, while running in a container on an EC2 instance:
- Bind mount the Unix Socket of the Docker Engine running on the host in to the running container, which permits the container full access to the underlying Docker API.
- Run a container in privileged mode, removing the Linux System Call and Security Module restrictions, allowing a Docker Engine to run in a container.
There are inherent risks involved in both of these approaches. Containers that have access to the host’s Docker daemon or run in privileged mode can also perform other malicious actions on the host. For example, a container with access to the host’s Docker Engine through a mounted Unix socket would have full access to the underlying Docker API. This would give the Container the privileges to start and stop any other container running on that Docker Engine, or even docker exec
into other containers. Even in single-tenant ECS clusters, this can lead to severe ramifications as it exposes a back door for hostile actors.
AWS Fargate runs each container in a VM-isolated environment. It also imposes security best practices, including prohibiting running containers from mounting directories or sockets from the underlying host and preventing containers from running with additional linux capabilities or using the --privileged
flag. As a result, customers cannot build container images inside Fargate containers using the builder within Docker Engine.
kaniko
New tools have emerged in the past few years to address the problem of building container images without requiring privileged mode. kaniko is one such tool that builds container images from a Dockerfile, much like the traditional Docker does. But unlike Docker, it doesn’t require root privileges, and it executes each command within a Dockerfile entirely in userspace. Thus, it permits you to build container images in rootless ways, such as in a running container. kaniko is an excellent standalone image builder, purposefully designed to run within a multi-tenant container cluster.
kaniko is designed to run within the constraints of a containerized environment, such as the one provided by Fargate. It does not require any additional Linux capabilities, for Linux Security Modules to be disabled, or any other access to the underlying host. It is, therefore, an ideal utility for building images on AWS Fargate.
In the next section, we will show you how to build container images in Fargate containers using kaniko.
Solution
To build images using kaniko with Amazon ECS on AWS Fargate, you would need:
- AWS CLI version 2
- Docker
- An existing AWS VPC and Subnet. The ECS tasks deployed in this subnet will also need access to your application’s source code repository. This could be a public GitHub repository; if so, the subnets would need internet access.
Let’s start by storing the IDs of the VPC and subnet you plan on using:
Create an ECS cluster:
Create an ECR repository to store the demo application.
Create an ECR repository to store the kaniko container image:
The upstream image provided by the kaniko community may work for you depending on your container repository. However, in this walk through, we need to pass a configuration file to allow kaniko to push to Amazon ECR. To push images to an ECR repository, the ECR Credential Helper will authenticate using AWS Credentials. The upstream kaniko container image already includes the ECR Credentials Helper binary. However, a configuration file is required to instruct kaniko to use the ECR Credential Helper for ECR authentication.
Build and push the image to ECR:
Create an IAM role for the ECS task that allows pushing the demo application’s container image to ECR:
Create an ECS task definition in which we define how the kaniko container will run, where the application source code repository is, and where to push the built container image:
Run kaniko as a single task using the ECS run-task API. This run-task API can be automated through a variety of CD and automation tools. If the subnet is a public subnet, the “assignPublicIp”
field should be set to “ENABLED”
.
Create a security group and create a kaniko task:
Once the task starts you can view kaniko logs using CloudWatch:
The task will build an image from source code. Once it pushes the image to ECR, the task will terminate.
List images in your ECR repository to verify that the built image has been pushed successfully:
Cleanup
Conclusion
With the increased security profile of AWS Fargate, customers leveraging traditional container image builders have been unable to take advantage of serverless compute and have been left provisioning and managing servers to support CD pipelines. In this blog post, we have shown how modern container image builders, such as kaniko, can run without additional Linux privileges in an Amazon ECS task running on AWS Fargate.
- To see how kaniko can be used in a Jenkins Pipeline on Amazon EKS, see this blog post.
- To learn more about kaniko, find additional documentation on their github repository.