Extending AWS CodeBuild with Custom Build Environments

AWS CodeBuild is a fully managed build service that compiles source code, runs tests, and produces software packages that are ready to deploy. CodeBuild provides curated build environments for programming languages and runtimes such as Java, Ruby, Python, Go, Node.js, Android, and Docker. It can be extended through the use of custom build environments to support many more.

Build environments are Docker images that include a complete file system with everything required to build and test your project. To use a custom build environment in a CodeBuild project, you build a container image for your platform that contains your build tools, push it to a Docker container registry such as Amazon EC2 Container Registry (ECR), and reference it in the project configuration. When building your application, CodeBuild will retrieve the Docker image from the container registry specified in the project configuration and use the environment to compile your source code, run your tests, and package your application.

In this post, we’ll create a build environment for PHP applications and walk through the steps to configure CodeBuild to use this environment.

Requirements

In order to follow this tutorial and build the Docker container image, you need to have the Docker platform, the AWS Command Line Interface, and Git installed.

Create the demo resources

To begin, we’ll clone codebuild-images from GitHub. It contains an AWS CloudFormation template that we’ll use to create resources for our demo: a source code repository in AWS CodeCommit and a Docker image repository in Amazon ECR. The repository also includes PHP sample code and tests that we’ll use to demonstrate our custom build environment.

1. Clone the Git repository:

   git clone https://github.com/awslabs/codebuild-images.git
   cd codebuild-images

2. Create the CloudFormation stack using the template.yml file. You can use the CloudFormation console to create the stack or you can use the AWS Command Line Interface:

   aws cloudformation create-stack \
    --stack-name codebuild-php \
    --parameters ParameterKey=EnvName,ParameterValue=php \
    --template-body file://template.yml > /dev/null && \
   aws cloudformation wait stack-create-complete \
    --stack-name codebuild-php && \
   aws cloudformation describe-stacks \
    --stack-name codebuild-php \
    --output table \
    --query Stacks[0].Outputs

After the stack has been created, CloudFormation will return two outputs:

• BuildImageRepositoryUri: the URI of the Docker repository that will host our build environment image.
• SourceCodeRepositoryCloneUrl: the clone URL of the Git repository that will host our sample PHP code.

Build and push the Docker image

Docker images are specified using a Dockerfile, which contains the instructions for assembling the image. The Dockerfile included in the PHP build environment contains these instructions:

FROM php:7

ARG composer_checksum=55d6ead61b29c7bdee5cccfb50076874187bd9f21f65d8991d46ec5cc90518f447387fb9f76ebae1fbbacf329e583e30
ARG composer_url=https://raw.githubusercontent.com/composer/getcomposer.org/ba0141a67b9bd1733409b71c28973f7901db201d/web/installer

ENV COMPOSER_ALLOW_SUPERUSER=1
ENV PATH=$PATH:vendor/bin

RUN apt-get update && apt-get install -y --no-install-recommends \
      curl \
      git \
      python-dev \
      python-pip \
      zlib1g-dev \
 && pip install awscli \
 && docker-php-ext-install zip \
 && curl -o installer "$composer_url" \
 && echo "$composer_checksum *installer" | shasum –c –a 384 \
 && php installer --install-dir=/usr/local/bin --filename=composer \
 && rm -rf /var/lib/apt/lists/*

This Dockerfile inherits all of the instructions from the official PHP Docker image, which installs the PHP runtime. On top of that base image, the build process will install Python, Git, the AWS CLI, and Composer, a dependency management tool for PHP. We’ve installed the AWS CLI and Git as tools we can use during builds. For example, using the AWS CLI, we could trigger a notification from Amazon Simple Notification Service (SNS) when a build is complete or we could use Git to create a new tag to mark a successful build. Finally, the build process cleans up files created by the packaging tools, as recommended in Best practices for writing Dockerfiles.

Next, we’ll build and push the custom build environment.

1. Provide authentication details for our registry to the local Docker engine by executing the output of the login helper provided by the AWS CLI:

   aws ecr get-login
   

2. Build and push the Docker image. We’ll use the repository URI returned in the CloudFormation stack output (BuildImageRepositoryUri) as the image tag:

   cd php
   docker build -t [BuildImageRepositoryUri] .
   docker push [BuildImageRepositoryUri]

After running these commands, your Docker image is pushed into Amazon ECR and ready to build your project.

Configure the Git repository

The repository we cloned includes a small PHP sample that we can use to test our PHP build environment. The sample function converts Roman numerals to Arabic numerals. The repository also includes a sample test to exercise this function. The sample also includes a YAML file called a build spec that contains commands and related settings that CodeBuild uses to run a build:

version: 0.1
phases:
  pre_build:
    commands:
      - composer install
  build:
    commands:
      - phpunit tests

This build spec configures CodeBuild to run two commands during the build:

• composer install to install configured dependencies, such as PHPUnit before the build.
• phpunit tests to run the unit tests during the build.

We will push the sample application to the CodeCommit repo created by the CloudFormation stack. You’ll need to grant your IAM user the required level of access to the AWS services required for CodeCommit and you’ll need to configure your Git client with the appropriate credentials. See Setup for HTTPS Users Using Git Credentials in the CodeCommit documentation for detailed steps.

We’re going to initialize a Git repository for our sample, configure our origin, and push the sample to the master branch in CodeCommit.

1. Initialize a new Git repository in the sample directory:

   cd sample
   git init

2. Add and commit the sample files to the repository:

   git add .
   git commit -m "Initial commit"

3. Configure the git remote and push the sample to it. We’ll use the repository clone URL returned in the CloudFormation stack output (SourceCodeRepositoryCloneUrl) as the remote URL:

   git remote add origin [SourceCodeRepositoryCloneUrl]
   git push origin master

Now that our sample application has been pushed into source control and our build environment image has been pushed into our Docker registry, we’re ready to create a CodeBuild project and start our first build.

Configure the CodeBuild project

In this section, we’ll walk through the steps for configuring CodeBuild to use the custom build environment.

1. In the AWS Management Console, open the AWS CodeBuild console, and then choose Create project.
2. In Project name, type php-demo.
3. From Source provider, choose AWS CodeCommit.  From Repository, choose codebuild-sample-php.
4. In Environment image, select Specify a Docker image. From Custom image type, choose Amazon ECR. From Amazon ECR Repository, choose codebuild/php.  From Amazon ECR image, choose latest.
5. In Build specification, select Use the buildspec.yml in the source code root directory.
6. In Artifacts type, choose No artifacts.
7. Choose Continue and then choose Save and Build.
8. On the next page, from Branch, choose master and then choose Start build.

CodeBuild will pull the build environment image from Amazon ECR and use it to test our application. CodeBuild will show us the status of each build step, the last 20 lines of log messages generated by the build process, and provide a link to Amazon CloudWatch Logs for more debugging output.

Summary

CodeBuild supports a number of platforms and languages out of the box. By using custom build environments, it can be extended to other runtimes. In this post, we built a PHP environment and demonstrated how to use it to test PHP applications.

We’re excited to see how customers extend and use CodeBuild to enable continuous integration and continuous delivery for their applications. Please leave questions or suggestions in the comments or share what you’ve learned extending CodeBuild for your own projects.