Creating multi-architecture Docker images to support Graviton2 using AWS CodeBuild and AWS CodePipeline

This post provides a clear path for customers who are evaluating and adopting Graviton2 instance types for performance improvements and cost-optimization.

Graviton2 processors are custom designed by AWS using 64-bit Arm Neoverse N1 cores. They power the T4g*, M6g*, R6g*, and C6g* Amazon Elastic Compute Cloud (Amazon EC2) instance types and offer up to 40% better price performance over the current generation of x86-based instances in a variety of workloads, such as high-performance computing, application servers, media transcoding, in-memory caching, gaming, and more.

More and more customers want to make the move to Graviton2 to take advantage of these performance optimizations while saving money.

During the transition process, a great benefit AWS provides is the ability to perform native builds for each architecture, instead of attempting to cross-compile on homogenous hardware. This has the benefit of decreasing build time as well as reducing complexity and cost to set up.

To see this benefit in action, we look at how to build a CI/CD pipeline using AWS CodePipeline and AWS CodeBuild that can build multi-architecture Docker images in parallel to aid you in evaluating and migrating to Graviton2.

Solution overview

With CodePipeline and CodeBuild, we can automate the creation of architecture-specific Docker images, which can be pushed to Amazon Elastic Container Registry (Amazon ECR). The following diagram illustrates this architecture.

The steps in this process are as follows:

1. Create a sample Node.js application and associated Dockerfile.
2. Create the buildspec files that contain the commands that CodeBuild runs.
3. Create three CodeBuild projects to automate each of the following steps:
   • CodeBuild for x86 – Creates a x86 Docker image and pushes to Amazon ECR.
   • CodeBuild for arm64 – Creates a Arm64 Docker image and pushes to Amazon ECR.
   • CodeBuild for manifest list – Creates a Docker manifest list, annotates the list, and pushes to Amazon ECR.
4. Automate the orchestration of these projects with CodePipeline.

Prerequisites

The prerequisites for this solution are as follows:

• The correct AWS Identity and Access Management (IAM) role permissions for your account allowing for the creation of the CodePipeline pipeline, CodeBuild projects, and Amazon ECR repositories
• An Amazon ECR repository named multi-arch-test
• A source control service such as AWS CodeCommit or GitHub that CodeBuild and CodePipeline can interact with
• The source code repository initialized and cloned locally

Creating a sample Node.js application and associated Dockerfile

For this post, we create a sample “Hello World” application that self-reports the processor architecture. We work in the local folder that is cloned from our source repository as specified in the prerequisites.

1. In your preferred text editor, add a new file with the following Node.js code:

// Hello World sample app.
const http = require('http');

const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end(`Hello World. This processor architecture is ${process.arch}`);
});

server.listen(port, () => {
  console.log(`Server running on processor architecture ${process.arch}`);
});

2. Save the file in the root of your source repository and name it app.js.
3. Commit the changes to Git and push the changes to our source repository. See the following code:

git add .
git commit -m "Adding Node.js sample application."
git push

We also need to create a sample Dockerfile that instructs the docker build command how to build the Docker images. We use the default Node.js image tag for version 14.

4. In a text editor, add a new file with the following code:

# Sample nodejs application
FROM node:14
WORKDIR /usr/src/app
COPY package*.json app.js ./
RUN npm install
EXPOSE 3000
CMD ["node", "app.js"]

5. Save the file in the root of the source repository and name it Dockerfile. Make sure it is Dockerfile with no extension.
6. Commit the changes to Git and push the changes to our source repository:

git add .
git commit -m "Adding Dockerfile to host the Node.js sample application."
git push

Creating a build specification file for your application

It’s time to create and add a buildspec file to our source repository. We want to use a single buildspec.yml file for building, tagging, and pushing the Docker images to Amazon ECR for both target native architectures, x86, and Arm64. We use CodeBuild to inject environment variables, some of which need to be changed for each architecture (such as image tag and image architecture).

A buildspec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build. For more information, see Build specification reference for CodeBuild.

The buildspec we add instructs CodeBuild to do the following:

• install phase – Update the yum package manager
• pre_build phase – Sign in to Amazon ECR using the IAM role assumed by CodeBuild
• build phase – Build the Docker image using the Docker CLI and tag the newly created Docker image
• post_build phase – Push the Docker image to our Amazon ECR repository

We first need to add the buildspec.yml file to our source repository.

1. In a text editor, add a new file with the following build specification:

version: 0.2
phases:
    install:
        commands:
            - yum update -y
    pre_build:
        commands:
            - echo Logging in to Amazon ECR...
            - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
    build:
        commands:
            - echo Build started on `date`
            - echo Building the Docker image...          
            - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
            - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG      
    post_build:
        commands:
            - echo Build completed on `date`
            - echo Pushing the Docker image...
            - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG

2. Save the file in the root of the repository and name it buildspec.yml.

Because we specify environment variables in the CodeBuild project, we don’t need to hard code any values in the buildspec file.

3. Commit the changes to Git and push the changes to our source repository:

git add .
git commit -m "Adding CodeBuild buildspec.yml file."
git push

Creating a build specification file for your manifest list creation

Next we create a buildspec file that instructs CodeBuild to create a Docker manifest list, and associate that manifest list with the Docker images that the buildspec file builds.

A manifest list is a list of image layers that is created by specifying one or more (ideally more than one) image names. You can then use it in the same way as an image name in docker pull and docker run commands, for example. For more information, see manifest create.

As of this writing, manifest creation is an experimental feature of the Docker command line interface (CLI).

Experimental features provide early access to future product functionality. These features are intended only for testing and feedback because they may change between releases without warning or be removed entirely from a future release. Experimental features must not be used in production environments. For more information, Experimental features.

When creating the CodeBuild project for manifest list creation, we specify a buildspec file name override as buildspec-manifest.yml. This buildspec instructs CodeBuild to do the following:

• install phase – Update the yum package manager
• pre_build phase – Sign in to Amazon ECR using the IAM role assumed by CodeBuild
• build phase – Perform three actions:
  • Set environment variable to enable Docker experimental features for the CLI
  • Create the Docker manifest list using the Docker CLI
  • Annotate the manifest list to add the architecture-specific Docker image references
• post_build phase – Push the Docker image to our Amazon ECR repository and use docker manifest inspect to echo out the contents of the manifest list from Amazon ECR

We first need to add the buildspec-manifest.yml file to our source repository.

1. In a text editor, add a new file with the following build specification:

version: 0.2
# Based on the Docker documentation, must include the DOCKER_CLI_EXPERIMENTAL environment variable
# https://docs.docker.com/engine/reference/commandline/manifest/    

phases:
    install:
        commands:
            - yum update -y
    pre_build:
        commands:
            - echo Logging in to Amazon ECR...
            - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
    build:
        commands:
            - echo Build started on `date`
            - echo Building the Docker manifest...   
            - export DOCKER_CLI_EXPERIMENTAL=enabled       
            - docker manifest create $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest-arm64v8 $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest-amd64    
            - docker manifest annotate --arch arm64 $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest-arm64v8
            - docker manifest annotate --arch amd64 $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest-amd64

    post_build:
        commands:
            - echo Build completed on `date`
            - echo Pushing the Docker image...
            - docker manifest push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME
            - docker manifest inspect $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME

2. Save the file in the root of the repository and name it buildspec-manifest.yml.
3. Commit the changes to Git and push the changes to our source repository:

git add .
git commit -m "Adding CodeBuild buildspec-manifest.yml file."
git push

Setting up your CodeBuild projects

Now we have created a single buildspec.yml file for building, tagging, and pushing the Docker images to Amazon ECR for both target native architectures: x86 and Arm64. This file is shared by two of the three CodeBuild projects that we create. We use CodeBuild to inject environment variables, some of which need to be changed for each architecture (such as image tag and image architecture). We also want to use the single Docker file, regardless of the architecture. We also need to ensure any third-party libraries are present and compiled correctly for the target architecture.

For more information about third-party libraries and software versions that have been optimized for Arm, see the Getting started with AWS Graviton GitHub repo.

We use the same environment variable names for the CodeBuild projects, but each project has specific values, as detailed in the following table. You need to modify these values to your numeric AWS account ID, the AWS Region where your Amazon ECR registry endpoint is located, and your Amazon ECR repository name. The instructions for adding the environment variables in the CodeBuild projects are in the following sections.

The image we use in this post uses architecture-specific tags with the term latest. This is for demonstration purposes only; it’s best to tag the images with an explicit version or another meaningful reference.

CodeBuild for x86

We start with creating a new CodeBuild project for x86 on the CodeBuild console.

CodeBuild looks for a file named buildspec.yml by default, unless overridden. For these first two CodeBuild projects, we rely on that default and don’t specify the buildspec name.

1. On the CodeBuild console, choose Create build project.
2. For Project name, enter a unique project name for your build project, such as node-x86.
3. To add tags, add them under Additional Configuration.
4. Choose a Source provider (for this post, we choose GitHub).
5. For Environment image, choose Managed image.
6. Select Amazon Linux 2.
7. For Runtime(s), choose Standard.
8. For Image, choose aws/codebuild/amazonlinux2-x86_64-standard:3.0.

This is a x86 build image.

9. Select Privileged.
10. For Service role, choose New service role.
11. Enter a name for the new role (one is created for you), such as CodeBuildServiceRole-nodeproject.

We reuse this same service role for the other CodeBuild projects associated with this project.

12. Expand Additional configurations and move to the Environment variables
13. Create the following Environment variables:

14. Choose Create build project.

Attaching the IAM policy

Now that we have created the CodeBuild project, we need to adjust the new service role that was just created and attach an IAM policy so that it can interact with the Amazon ECR API.

1. On the CodeBuild console, choose the node-x86 project
2. Choose the Build details
3. Under Service role, choose the link that looks like arn:aws:iam::111111111111:role/service-role/CodeBuildServiceRole-nodeproject.

A new browser tab should open.

4. Choose Attach policies.
5. In the Search field, enter AmazonEC2ContainerRegistryPowerUser.
6. Select AmazonEC2ContainerRegistryPowerUser.
7. Choose Attach policy.

CodeBuild for arm64

Now we move on to creating a new (second) CodeBuild project for Arm64.

1. On the CodeBuild console, choose Create build project.
2. For Project name, enter a unique project name, such as node-arm64.
3. If you want to add tags, add them under Additional Configuration.
4. Choose a Source provider (for this post, choose GitHub).
5. For Environment image, choose Managed image.
6. Select Amazon Linux 2.
7. For Runtime(s), choose Standard.
8. For Image, choose aws/codebuild/amazonlinux2-aarch64-standard:2.0.

This is an Arm build image and is different from the image selected in the previous CodeBuild project.

9. Select Privileged.
10. For Service role, choose Existing service role.
11. Choose CodeBuildServiceRole-nodeproject.
12. Select Allow AWS CodeBuild to modify this service role so it can be used with this build project.
13. Expand Additional configurations and move to the Environment variables
14. Create the following Environment variables:

15. Choose Create build project.

CodeBuild for manifest list

For the last CodeBuild project, we create a Docker manifest list, associating that manifest list with the Docker images that the preceding projects create, and pushing the manifest list to ECR. This project uses the buildspec-manifest.yml file created earlier.

1. On the CodeBuild console, choose Create build project.
2. For Project name, enter a unique project name for your build project, such as node-manifest.
3. If you want to add tags, add them under Additional Configuration.
4. Choose a Source provider (for this post, choose GitHub).
5. For Environment image, choose Managed image.
6. Select Amazon Linux 2.
7. For Runtime(s), choose Standard.
8. For Image, choose aws/codebuild/amazonlinux2-x86_64-standard:3.0.

This is a x86 build image.

9. Select Privileged.
10. For Service role, choose Existing service role.
11. Choose CodeBuildServiceRole-nodeproject.
12. Select Allow AWS CodeBuild to modify this service role so it can be used with this build project.
13. Expand Additional configurations and move to the Environment variables
14. Create the following Environment variables:

15. For Buildspec name – optional, enter buildspec-manifest.yml to override the default.
16. Choose Create build project.

Setting up CodePipeline

Now we can move on to creating a pipeline to orchestrate the builds and manifest creation.

1. On the CodePipeline console, choose Create pipeline.
2. For Pipeline name, enter a unique name for your pipeline, such as node-multi-architecture.
3. For Service role, choose New service role.
4. Enter a name for the new role (one is created for you). For this post, we use the generated role name CodePipelineServiceRole-nodeproject.
5. Select Allow AWS CodePipeline to create a service role so it can be used with this new pipeline.
6. Choose Next.
7. Choose a Source provider (for this post, choose GitHub).
8. If you don’t have any existing Connections to GitHub, select Connect to GitHub and follow the wizard.
9. Choose your Branch name (for this post, I choose main, but your branch might be different).
10. For Output artifact format, choose CodePipeline default.
11. Choose Next.

You should now be on the Add build stage page.

12. For Build provider, choose AWS CodeBuild.
13. Verify the Region is your Region of choice (for this post, I use US East (N. Virginia)).
14. For Project name, choose node-x86.
15. For Build type, select Single build.
16. Choose Next.

You should now be on the Add deploy stage page.

17. Choose Skip deploy stage.

A pop-up appears that reads Your pipeline will not include a deployment stage. Are you sure you want to skip this stage?

18. Choose Skip.
19. Choose Create pipeline.

CodePipeline immediately attempts to run a build. You can let it continue without worry if it fails. We are only part of the way done with the setup.

Adding an additional build step

We need to add the additional build step for the Arm CodeBuild project in the Build stage.

1. On the CodePipeline console, choose node-multi-architecture pipeline
2. Choose Edit to start editing the pipeline stages.

You should now be on the Editing: node-multi-architecture page.

3. For the Build stage, choose Edit stage.
4. Choose + Add action.

5. For Action name, enter Build-arm64.
6. For Action provider, choose AWS CodeBuild.
7. Verify your Region is correct.
8. For Input artifacts, select SourceArtifact.
9. For Project name, choose node-arm64.
10. For Build type, select Single build.
11. Choose Done.
12. Choose Save.

A pop-up appears that reads Saving your changes cannot be undone. If the pipeline is running when you save your changes, that execution will not complete.

13. Choose Save.

Updating the first build action name

This step is optional. The CodePipeline wizard doesn’t allow you to enter your Build action name during creation, but you can update the Build stage’s first build action to have consistent naming.

1. Choose Edit to start editing the pipeline stages.
2. Choose the Edit icon.
3. For Action name, enter Build-x86.
4. Choose Done.
5. Choose Save.

A pop-up appears that says Saving your changes cannot be undone. If the pipeline is running when you save your changes, that execution will not complete.

6. Choose Save.

Adding the project

Now we add the CodeBuild project for manifest creation and publishing.

1. On the CodePipeline console, choose node-multi-architecture pipeline.
2. Choose Edit to start editing the pipeline stages.
3. Choose +Add stage below the Build
4. Set the Stage name to Manifest
5. Choose +Add action group.
6. For Action name, enter Create-manifest.
7. For Action provider, choose AWS CodeBuild.
8. Verify your Region is correct.
9. For Input artifacts, select SourceArtifact.
10. For Project name, choose node-manifest.
11. For Build type, select Single build.
12. Choose Done.
13. Choose Save.

A pop-up appears that reads Saving your changes cannot be undone. If the pipeline is running when you save your changes, that execution will not complete.

14. Choose Save.

Testing the pipeline

Now let’s verify everything works as planned.

1. In the pipeline details page, choose Release change.

This runs the pipeline in stages. The process should take a few minutes to complete. The pipeline should show each stage as Succeeded.

Now we want to inspect the output of the Create-manifest action that runs the CodeBuild project for manifest creation.

2. Choose Details in the Create-manifest

This opens the CodeBuild pipeline.

3. Under Build logs, we should see the output from the manifest inspect command we ran as the last step in the buildspec-manifest.yml See the following sample log:

[Container] 2020/10/07 16:47:39 Running command docker manifest inspect $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1369,
         "digest": "sha256:238c2762212ff5d7e0b5474f23d500f2f1a9c851cdd3e7ef0f662efac508cd04",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1369,
         "digest": "sha256:0cc9e96921d5565bdf13274e0f356a139a31d10e95de9ad3d5774a31b8871b05",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Cleaning up

To avoid incurring future charges, clean up the resources created as part of this post.

1. On the CodePipeline console, choose the pipeline node-multi-architecture.
2. Choose Delete pipeline.
3. When prompted, enter delete.
4. Choose Delete.
5. On the CodeBuild console, choose the Build project node-x86.
6. Choose Delete build project.
7. When prompted, enter delete.
8. Choose Delete.
9. Repeat the deletion process for Build projects node-arm64 and node-manifest.

Next we delete the Docker images we created and pushed to Amazon ECR. Be careful to not delete a repository that is being used for other images.

10. On the Amazon ECR console, choose the repository multi-arch-test.

You should see a list of Docker images.

11. Select latest, latest-arm64v8, and latest-amd64.
12. Choose Delete.
13. When prompted, enter delete.
14. Choose Delete.

Finally, we remove the IAM roles that we created.

15. On the IAM console, choose Roles.
16. In the search box, enter CodePipelineServiceRole-nodeproject.
17. Select the role and choose Delete role.
18. When prompted, choose Yes, delete.
19. Repeat these steps for the role CodeBuildServiceRole-nodeproject.

Conclusion

To summarize, we successfully created a pipeline to create multi-architecture Docker images for both x86 and arm64. We referenced them via annotation in a Docker manifest list and stored them in Amazon ECR. The Docker images were based on a single Docker file that uses environment variables as parameters to allow for Docker file reuse.

For more information about these services, see the following:

• Getting started with AWS Graviton
• Docker sample for CodeBuild
• Pushing a multi-architecture image
• Amazon ECR Managed Policies
• Docker manifest documentation

About the Authors

Tyler Lynch
Tyler Lynch is a Sr. Solutions Architect focusing on EdTech at AWS.

Alistair McLean

Alistair is a Principal Solutions Architect focused on State and Local Government and K12 customers at AWS.