Reduce Docker image build time on AWS CodeBuild using Amazon ECR as a remote cache

In modern software development, containerization with Docker has revolutionized how we build and deploy applications. While Docker enables packaging applications into portable containers, the continuous need to update these images can be resource intensive. AWS CodeBuild addresses this challenge by providing a managed build service that eliminates infrastructure maintenance overhead. In this blog post, we’ll explore how AWS CodeBuild integration with Amazon Elastic Container Registry (Amazon ECR) as a cache backend can significantly accelerate our Docker image build process, making development more efficient and streamlined.

AWS CodeBuild creates isolated environments for each build, which means build artifacts cannot be permanently stored on the host system. While CodeBuild does offer a native local caching feature, it provides only temporary storage and is most effective for builds that occur in quick succession.

This local caching mechanism, however, is not reliable when builds are triggered at varying intervals, as it operates on a best-effort basis. To address this limitation, we recommend using Amazon Elastic Container Registry as a persistent cache for Docker layers. This solution offers several advantages:

• It provides a reliable, long-term storage solution for build caches
• The cached layers can be reused across multiple builds regardless of timing
• The cache remains valid and accessible at any point in time

This post shows how to implement a simple, effective, and durable Docker layer cache for CodeBuild using Amazon ECR repository as a cache backend to significantly reduce image build runtime.

Solution Overview

The following diagram illustrates the high-level architecture of this solution. We describe implementing each stage in more detail in the following paragraphs.

Figure 1: Solution Flow Diagram

To use an Amazon ECR registry as a backend for caching, we must first enable the containerd image store in our Docker driver. This feature is not enabled in the default Docker driver configuration. Therefore, we create a new docker driver using docker buildx command with containerd (docker-container driver) image store enabled.

When CodeBuild runs for the first time, it will attempt to retrieve cache data from the Amazon ECR repository. Since this is the first run, no cache will be available. CodeBuild will then proceed to build the Docker image from scratch, generate cache data during this initial build and export both the newly built image and its associated cache to the Amazon ECR repository.

In each subsequent build, CodeBuild will import the previously stored cache from Amazon ECR. This cached data will be used to speed up the image building process, as only the changed layers will need to be rebuilt. Finally, the updated cache and image will be stored back in Amazon ECR.

Prerequisites

Before we begin the walk-through, we must have an AWS account. If you don’t have one, sign up at https://aws.amazon.com.

Walk-through

Launch the following AWS CloudFormation template to create Amazon ECR repository and AWS CodeBuild project including CodeBuild service role and required permission as a managed policy.

AWSTemplateFormatVersion: "2010-09-09"

Description: 'AWS CloudFormation template to create infrastructure which demo using Amazon ECR as a remote cache for AWS CodeBuild'

Parameters:
  CodeBuildProjectName:
    Type: String
    Default: CBECRCacheDemoProject
    Description: "Enter name for your CodeBuild project"
  CodeBuildServiceRolePolicyName:
    Type: String
    Default: CodeBuildDockerCachePolicy
    Description: "Enter name for the IAM policy"
  ECRRepoName:
    Type: String
    Default: amazon_linux_codebuild_image
    Description: "Enter name for Amazon ECR repository"
  GitHubLocation:
    Type:  String
    Default: "https://github.com/aws/aws-codebuild-docker-images"
    Description: "Enter your source code GitHub URL"
  ImageTag:
    Type: String
    Default: demo
    Description: "Enter Tag name for your application docker image"
  CacheTag:
    Type: String
    Default: demo-cache
    Description: "Enter tag name for the cache image"

Resources:

  CodeBuildServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument: |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "codebuild.amazonaws.com"
               },
              "Action": "sts:AssumeRole"
            }
         ]
        }
      Path: /

  CodeBuildServiceRolePolicy:
    Type: AWS::IAM::RolePolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - ecr:BatchGetImage
              - ecr:BatchCheckLayerAvailability
              - ecr:InitiateLayerUpload
              - ecr:UploadLayerPart
              - ecr:CompleteLayerUpload
              - ecr:PutImage
              - ecr:GetDownloadUrlForLayer
            Resource: !GetAtt ECRRepository.Arn
          - Effect: Allow
            Action: 
              - ecr:GetAuthorizationToken
            Resource: '*'
          - Effect: Allow
            Action:
              - codeconnections:UseConnection
              - codeconnections:GetConnectionToken
              - codeconnections:GetConnection
              - codestar-connections:GetConnectionToken
              - codestar-connections:GetConnection
            Resource: '*'
          - Effect: Allow
            Action: 
              - logs:CreateLogStream
              - logs:CreateLogGroup
              - logs:PutLogEvents
            Resource: 
              - !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/${CodeBuildProjectName}'
              - !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/${CodeBuildProjectName}:*'
          - Effect: Allow
            Action:
              - s3:PutObject
              - s3:GetObject
              - s3:GetObjectVersion
              - s3:GetBucketAcl
              - s3:GetBucketLocation
            Resource: 
              - !Sub 'arn:${AWS::Partition}:s3:::codepipeline-${AWS::Region}-*'
      PolicyName: !Ref CodeBuildServiceRolePolicyName
      RoleName: !Ref CodeBuildServiceRole

  ECRRepository:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: !Ref ECRRepoName
      ImageScanningConfiguration:
        ScanOnPush: true

  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Ref CodeBuildProjectName
      Source:
        Type: GITHUB
        Location: !Ref GitHubLocation
        BuildSpec: !Sub |
          version: 0.2

          phases:
            install:
             commands:
               - docker buildx create --name containerd --driver=docker-container --driver-opt default-load=true

            pre_build:
             commands:
               - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin ${AWS::AccountId}.dkr.ecr.$AWS_REGION.amazonaws.com

            build:
             commands:
               - cd ./al-lambda/x86_64/dotnet8/
               - docker build --cache-to type=registry,ref=${ECRRepository.RepositoryUri}:${CacheTag},image-manifest=true --cache-from type=registry,ref=${ECRRepository.RepositoryUri}:${CacheTag} --tag ${ECRRepository.RepositoryUri}:${ImageTag} --builder=containerd .

            post_build:
             commands:
               - docker push ${ECRRepository.RepositoryUri}:${ImageTag}
      ServiceRole: !GetAtt CodeBuildServiceRole.Arn
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        Image: aws/codebuild/amazonlinux-x86_64-standard:5.0
        ComputeType: BUILD_GENERAL1_SMALL
        PrivilegedMode: true
      Cache:
        Type: LOCAL
        Modes:
          - LOCAL_DOCKER_LAYER_CACHE

Specify the following parameters while creating the CloudFormation stack (see Figure 2):

1.  Set the cache image tag (CacheTag) to “demo-cache“.
2. Name the CodeBuild project (CodeBuildProjectName) as “CBECRCacheDemoProject“.
3. Specify the IAM policy name (CodeBuildServiceRolePolicyName) as “CodeBuildDockerCachePolicy“.
4. Define the ECR repository name (ECRRepoName) as “amazon_al_lambda_codebuild_image“.
5. Enter the GitHub repository URL (GitHubLocation) as “https://github.com/aws/aws-codebuild-docker-images“.
6. Set the Docker image tag (ImageTag) to “demo“

Figure 2: CloudFormation Stack parameter

The CloudFormation stack will set up a comprehensive development environment for our project. It will create a CodeBuild project equipped with all necessary IAM roles and permissions, ensuring smooth and secure build processes. Additionally, the stack will create an Amazon ECR repository. This repository is configured to automatically scan Docker images for vulnerabilities upon upload. The ECR will serve as a secure storage location for both our Docker images and cache images.

The CodeBuild project will be created with a buildspec file which will instruct CodeBuild to do the following:

• Creates a new driver called “containerd” using buildx since the default Docker driver supports registry cache backend only when the containerd image store is enabled.
• To pull and push both Docker images and cache, authentication with the Amazon ECR repository is required.
• During the image build process:
  • We use the --cache-from parameter to force Docker to check for and use any existing cache from the repository.
  • The image-manifest option is set to true to enable cache storage in the Amazon ECR repository.
  • The --cache-to parameter is used to push or update the cache to the Amazon ECR repository.
• After the build is complete, in the post-build phase, the image is pushed to Amazon ECR. The cache is automatically uploaded to the Amazon ECR repository as part of the Docker build command execution.

Testing the solution

After having successfully created the CloudFormation stack, we can proceed to test and evaluate how it performs.

The initial build process took approximately 10 minutes to complete. Since this was the first execution, no cache was available in Amazon ECR, requiring the system to build the image entirely from scratch. Although the cache import operation failed as expected during this initial run, the system continued with the build process without any cached layers. This first build served a dual purpose: it not only created the application image but also generated a cache, which was then exported to Amazon ECR as a separate image. This cached data would become available for future builds, setting the foundation for more efficient subsequent builds.

To verify the effectiveness of the solution, we triggered a second build after introducing a minor modification – adding an echo command in the middle of the Dockerfile’s active commands (excluding commented line). During this subsequent build, Docker intelligently utilized the cached layers up until the point of modification, after which it rebuilt only the necessary layers. This smart caching strategy resulted in a build time of approximately 6 minutes, clearly demonstrating how the caching system optimizes the build process even when changes are introduced. Further validation across multiple large-scale projects confirmed the effectiveness of this approach, consistently achieving build time reductions of up to 25%.

We enabled CodeBuild’s built-in Docker layer caching feature on a best-effort basis. This approach is recommended as it uses cached layers in the local when available instead of downloading them from the repository, which will further improve the overall build speed.

Cleaning up

When we finished testing, we should de-provision the following resources to avoid incurring further charges and keep the account clean from unused resources:

• Delete the docker images from the Amazon ECR repository amazon_al_lambda_codebuild_image.
• Delete the CloudFormation stack which has been created in the “Launch the AWS CloudFormation template” section.

Conclusion

In this discussion, we explored an efficient and straightforward solution for implementing external Docker caching in CodeBuild using Amazon ECR as a backend storage system. This approach offers several key benefits:

The solution reduces Docker build times in CodeBuild up to 25% and is versatile enough to handle most scenarios, including complex multi-stage builds. A particularly valuable advantage is that Amazon ECR stores the cache separately in its repository, making it reusable across different projects.

The business impact is substantial: shorter build times lead directly to reduced compute costs. More importantly, this optimization results in a more streamlined development lifecycle, enabling faster feature releases at lower operational costs.

In essence, this caching solution not only improves technical efficiency but also delivers tangible business value through reduced costs and accelerated development cycles.

About the author