How to trigger AWS CodeBuild jobs for selective file changes in AWS CodeCommit

Unnecessary build pipelines cause unnecessary CPU and storage consumption. If you’re managing your application source code and your documentation files in the same repository, you might want the flexibility to build only the files that have changed. For example, the application doesn’t need to be rebuilt every time the documentation files change.

Version control process is an important requirement in the software development model. Continuous integration (CI) is a DevOps software development practice where developers regularly merge their code changes into a central repository. After a merge, automated builds and tests are run. The key goals of CI are to find and address bugs quicker, improve software quality, and reduce the time it takes to validate and release software updates.

App teams often require pipeline build times to be shorter so product features can be delivered faster and aligned with the business needs. They usually manage their application source code (like files with extensions .py, .java, .c, etc.) along with the required documentation files (like files with extensions .md, .docx, .pdf, etc.) in the same repository. A build job corresponding to this repository builds and publishes the source code and the documentation files as artifacts for deployment. Any change in the documentation files doesn’t change the functionality of the application code, but it still triggers an unnecessary build pipeline.

In this blog post, we show you how to control and restrict the build process to selective file changes in AWS CodeCommit with AWS Lambda and AWS CodeBuild.

Solution scenario

Consider a scenario in which an CodeCommit repository has a Dockerfile and documentation files. You can use the CodeBuild job to build and publish a new Docker image to Amazon Elastic Container Registry (Amazon ECR) by identifying the modification to the Dockerfile with the help of a Lambda function.

Solution walkthrough

The following diagram shows how AWS services are used for this demonstration.

You can configure the CodeCommit repository so that code pushes, or other events, trigger actions, such as sending a notification from Amazon Simple Notification Service (Amazon SNS) or invoking a function in Lambda. You can create up to 10 triggers for each CodeCommit repository. The solution involves the following steps.

1. The developers of an application make changes to the source code or the documentation.
2. The CodeCommit repository is configured so that code pushes or other events trigger actions, such as invoking a Lambda function.
3. The Lambda function performs the following steps:
   1. Uses the GetCommit operation in the CodeCommit API to get the latest commit.
   2. Uses the GetDifferences operation to get a list of each file that was added, modified, or deleted.
   3. From the list of files identified above, checks whether the Dockerfile is present.
   4. If yes, then triggers the CodeBuild job to build the Docker image. If no, then no action is taken.
4. The CodeBuild job builds the Docker image based on the Dockerfile and pushes the created image to the Amazon ECR repository.

The Lambda code is written in Python using the Boto3 libraries. You can find the example source code in aws-codecommit-selective-build-trigger repository.

Provisioning the AWS resources

It is time to deploy the solution using an AWS CloudFormation template.

Setting up

1. Download the source code from the aws-codecommit-selective-build-trigger GitHub repository.
2. Sign in to the AWS Management Console and choose the AWS Region of your choice.
3. Create an Amazon Simple Storage Service (Amazon S3) bucket. For information, see How Do I Create an S3 Bucket? in the AWS documentation.
4. Upload the Lambda deployment package, lambda.zip, to the S3 bucket that you created.

For this example, we created an S3 bucket named codecommit-selective-build in the N. Virginia (us-east-1) Region and used the console to upload the deployment package from the GitHub repository to the S3 bucket.

Creating the stack

1. In the AWS CloudFormation console, choose Create stack.
2. On the Specify template page, choose Upload a template file, and then choose the aws-codecommit-selective-build-trigger.yml template, downloaded from the GitHub repository.INSERT IMAGE
3. Specify the following parameters:
   • Stack name: CodecommitSelectiveBuildStack (You can use your own stack name, if you prefer.)
   • LambdaZipS3Bucket: codecommit-selective-build (This is the name of the S3 bucket where you put the sample code.)
   • LambdaZipS3Key: lambda.zip (This is the key name of the sample code S3 object. The object should be a zip file.)
   • ProjectName: sampletest (The name of your CodeCommit repository.)

   

4. Choose Next and again Next.
5. On the Review page, under Capabilities, choose the following option:I acknowledge that AWS CloudFormation might create IAM resources with custom names.
   
6. Choose Create stack to begin the stack creation process. When the stack creation is complete, the resources that were created are displayed on the Resources tab. The stack creation takes approximately 5-7 minutes.

The stack creates an CodeCommit repository (sampletestcoderepo), which is configured for invoking a Lambda function (sampletest-CodeBuild-Job-Lambda-Trigger) when commits are pushed to it.

Testing the solution

Now, you can try to push a commit to the remote CodeCommit repository.

1. Clone the CodeCommit repository sampletestcoderepo to your local computer. For information, see Connect to an AWS CodeCommit Repository in the AWS documentation. The following example shows how to clone a repository named sampletestcoderepo in the US East (N. Virginia) Region.

   git clone codecommit::us-east-1://sampletestcoderepo

    

2. Enter the cloned folder and create a README.md file.

   cd sampletestcoderepo
   echo -e "# sampletest-CodeRepo\nTest Repo" > README.md
   

    

3. Add and commit this file change. Committing this file does not trigger a build process.

   git add README.md
   git commit -m 'Create documentation file'

    

4. Push the commit to the remote CodeCommit repository.

   git push

    

   Look for the following output.

   Counting objects: 100% (3/3), done.
   Writing objects: 100% (3/3), 235 bytes | 235.00 KiB/s, done.
   
   …
   * [new branch]      master -> master

    

5. Now, to trigger the build, create a basic Dockerfile.

   echo -e "FROM busybox:latest\nMAINTAINER XXX" > Dockerfile

    

6. Add and commit the following file change.

   git add Dockerfile
   git commit -m 'Create initial source code file'

    

7. Push the commit to the remote CodeCommit repository.

   git push

    

After the local commit has been pushed to the remote CodeCommit repository sampletestcoderepo, the Lambda sampletest-CodeBuild-Job-Lambda-Trigger function detects the file addition, and it triggers the CodeBuild job sampletest-CodeBuild-Job. The CodeBuild job processes the Dockerfile and creates the Docker image, which is then pushed to the Amazon ECR repository sampletest-imagerepo. The following screenshot shows a sample CodeBuild run output.

Cleanup

To avoid additional infrastructure costs from the examples described in this post, be sure to delete the stack. On the AWS CloudFormation console, select the stack that you created, and then choose Delete. This automatically deletes the resources that it created, including CodeCommit repositories, IAM roles/policies, Lambda functions, and CodeBuild projects.

Conclusion

In this blog post, we showed you how to build a sample solution that triggers build jobs based on the files that are modified in the CodeCommit repository. Thus, the solution restricts unnecessary builds from getting triggered, which avoids unnecessary CPU and storage consumption.

You can find the example CloudFormation template and Lambda function, demonstrated in this blog post, in the aws-codecommit-selective-build-trigger GitHub repository. Using the sample code, you can customize the Lambda code to trigger CodeBuild job for specific files or file extensions.

If you have questions or other feedback about this example, please open an issue or submit a pull request.