AWS News Blog
New – Building a Continuous Integration Workflow with Step Functions and AWS CodeBuild
|
May 29, 2020: Post updated to include AWS CodePipeline support to invoke Step Functions with a new action type.
Automating your software build is an important step to adopt DevOps best practices. To help you with that, we built AWS CodeBuild, a fully managed continuous integration service that compiles source code, runs tests, and produces packages that are ready for deployment, and AWS CodePipeline, a fully managed continuous delivery service to automate your release pipelines.
However, there are so many possible customizations in our customers’ build processes, and we have seen developers spend time in creating their own custom workflows to coordinate the different activities required by their software build. For example, you may want to run, or not, some tests, or skip static analysis of your code when you need to deploy a quick fix. Depending on the results of your unit tests, you may want to take different actions, or be notified via SNS.
CodePipeline is currently optimized for releasing software and infrastructure in production. It includes release safety features that don’t exist in general workflow automation tools. On the other side, CodePipeline isn’t currently optimized for non-release scenarios such as orchestrating validation of a change prior to release, or for running multiple builds in parallel. To simplify those use cases, we are launching today a new AWS Step Functions service integration with CodeBuild. Now, during the execution of a state machine, you can start or stop a build, get build report summaries, and delete past build executions records.
In this way, you can define your own workflow-driven build process, and trigger it manually or automatically. For example you can:
- Use the new CodePipeline support to invoke Step Functions to customize your delivery pipeline with choices, external validations, or parallel tasks. Each of those tasks can now call CodeBuild to create a custom build following specific requirements. In this way, you can easily use Step Functions and CodePipeline together. For a comparison of when you should use Step Functions vs CodePipeline for managing deployments, please see this FAQs.
- Use Amazon EventBridge rules to start the build workflow periodically (for nightly builds) or when something happens (such as a a pull request to an AWS CodeCommit repository).
- Build a webhook that can be called by services such as GitHub using the Amazon API Gateway, either with a direct integration to a state machine, or via a AWS Lambda function that checks the validity of the input payload before starting the workflow.
With this integration, you can use the full capabilities of Step Functions to automate your software builds. For example, you can use a Parallel state to create parallel builds for independent components of the build. Starting from a list of all the branches in your code repository, you can use a Map state to run a set of steps (automating build, unit tests, and integration tests) for each branch. You can also leverage in the same workflow other Step Functions service integrations. For instance, you can send a message to an Amazon SQS queue to track your activities, or start a containerized application you just built using Amazon Elastic Container Service (Amazon ECS) and AWS Fargate.
Using Step Functions for a Workflow-Driven Build Process
I am working on a Java web application. To be sure that it works as I add new features, I wrote a few tests using JUnit Jupiter. I want those tests to be run just after the build process, but not always because tests can slow down some quick iterations. When I run tests, I want to store and view the reports of my tests using CodeBuild. At the end, I want to be notified in an SNS topic if the tests run, and if they were successful.
I created a repository in CodeCommit and I included two buildspec files for CodeBuild:
-
buildspec.ymlis the default and is using Apache Maven to run the build and the tests, and then is storing test results as reports.
version: 0.2
phases:
build:
commands:
- mvn package
artifacts:
files:
- target/binary-converter-1.0-SNAPSHOT.jar
reports:
SurefireReports:
files:
- '**/*'
base-directory: 'target/surefire-reports'-
buildspec-notests.ymlis doing only the build, and no tests are executed.
version: 0.2
phases:
build:
commands:
- mvn package -DskipTests
artifacts:
files:
- target/binary-converter-1.0-SNAPSHOT.jar
To set up the CodeBuild project and the Step Functions state machine to automate the build, I am using AWS CloudFormation with the following template:
AWSTemplateFormatVersion: 2010-09-09
Description: AWS Step Functions sample project for getting notified on AWS CodeBuild test report results
Resources:
CodeBuildStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
RoleArn: !GetAtt [ CodeBuildExecutionRole, Arn ]
DefinitionString:
!Sub
- |-
{
"Comment": "An example of using CodeBuild to run (or not run) tests, get test results and send a notification.",
"StartAt": "Run Tests?",
"States": {
"Run Tests?": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.tests",
"BooleanEquals": false,
"Next": "Trigger CodeBuild Build Without Tests"
}
],
"Default": "Trigger CodeBuild Build With Tests"
},
"Trigger CodeBuild Build With Tests": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::codebuild:startBuild.sync",
"Parameters": {
"ProjectName": "${projectName}"
},
"Next": "Get Test Results"
},
"Trigger CodeBuild Build Without Tests": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::codebuild:startBuild.sync",
"Parameters": {
"ProjectName": "${projectName}",
"BuildspecOverride": "buildspec-notests.yml"
},
"Next": "Notify No Tests"
},
"Get Test Results": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::codebuild:batchGetReports",
"Parameters": {
"ReportArns.$": "$.Build.ReportArns"
},
"Next": "All Tests Passed?"
},
"All Tests Passed?": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.Reports[0].Status",
"StringEquals": "SUCCEEDED",
"Next": "Notify Success"
}
],
"Default": "Notify Failure"
},
"Notify Success": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::sns:publish",
"Parameters": {
"Message": "CodeBuild build tests succeeded",
"TopicArn": "${snsTopicArn}"
},
"End": true
},
"Notify Failure": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::sns:publish",
"Parameters": {
"Message": "CodeBuild build tests failed",
"TopicArn": "${snsTopicArn}"
},
"End": true
},
"Notify No Tests": {
"Type": "Task",
"Resource": "arn:${AWS::Partition}:states:::sns:publish",
"Parameters": {
"Message": "CodeBuild build without tests",
"TopicArn": "${snsTopicArn}"
},
"End": true
}
}
}
- {snsTopicArn: !Ref SNSTopic, projectName: !Ref CodeBuildProject}
SNSTopic:
Type: AWS::SNS::Topic
CodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
ServiceRole: !Ref CodeBuildServiceRole
Artifacts:
Type: NO_ARTIFACTS
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:2.0
Source:
Type: CODECOMMIT
Location: https://git-codecommit.us-east-1.amazonaws.com/v1/repos/binary-converter
CodeBuildExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "sts:AssumeRole"
Principal:
Service: states.amazonaws.com
Path: "/"
Policies:
- PolicyName: CodeBuildExecutionRolePolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "sns:Publish"
Resource:
- !Ref SNSTopic
- Effect: Allow
Action:
- "codebuild:StartBuild"
- "codebuild:StopBuild"
- "codebuild:BatchGetBuilds"
- "codebuild:BatchGetReports"
Resource: "*"
- Effect: Allow
Action:
- "events:PutTargets"
- "events:PutRule"
- "events:DescribeRule"
Resource:
- !Sub "arn:${AWS::Partition}:events:${AWS::Region}:${AWS::AccountId}:rule/StepFunctionsGetEventForCodeBuildStartBuildRule"
CodeBuildServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "sts:AssumeRole"
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Path: /
Policies:
- PolicyName: CodeBuildServiceRolePolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
- "codebuild:CreateReportGroup"
- "codebuild:CreateReport"
- "codebuild:UpdateReport"
- "codebuild:BatchPutTestCases"
- "codecommit:GitPull"
Resource: "*"
Outputs:
StateMachineArn:
Value: !Ref CodeBuildStateMachine
ExecutionInput:
Description: Sample input to StartExecution.
Value:
>
{}When the CloudFormation stack has been created, there are two CodeBuild tasks in the state machine definition:
- The first CodeBuild task is using a synchronous integration (
startBuild.sync) to automatically wait for the build to terminate before progressing to the next step:
"Trigger CodeBuild Build With Tests": {
"Type": "Task",
"Resource": "arn:aws:states:::codebuild:startBuild.sync",
"Parameters": {
"ProjectName": "CodeBuildProject-HaVamwTeX8kM"
},
"Next": "Get Test Results"
}- The second CodeBuild task is using the
BuildspecOverrideparameter to override the default buildspec file used by the build with the one not running tests:
"Trigger CodeBuild Build Without Tests": {
"Type": "Task",
"Resource": "arn:aws:states:::codebuild:startBuild.sync",
"Parameters": {
"ProjectName": "CodeBuildProject-HaVamwTeX8kM",
"BuildspecOverride": "buildspec-notests.yml"
},
"Next": "Notify No Tests"
},
The first step is a Choice that looks into the input of the state machine execution to decide if to run tests, or not. For example, to run tests I can give in input:
{
"tests": true
}This is the visual workflow of the execution running tests, all tests are passed.
I change the value of "tests" to false, and start a new execution that goes on a different branch.
This time the buildspec is not executing tests, and I get a notification that no tests were run.
When starting this workflow automatically after an activity on GitHub or CodeCommit, I could look into the last commit message for specific patterns, and customize the build process accordingly. For example, I could skip tests if the [skip tests] string is part of the commit message. Similarly, in a production environment I could skip code static analysis, to have faster integration for urgent changes, if the [skip static analysis] message in included in the commit.
Extending the Workflow for Containerized Applications
A great way to distribute applications to different environments, is to package them as Docker images. In this way, I can also add a step to my build workflow and start the containerized application in an Amazon Elastic Container Service (Amazon ECS) task (running on AWS Fargate) for the Quality Assurance (QA) team.
First, I create an image repository in ECR and add permissions to the service role used by the CodeBuild project to upload to ECR, as described here.
Then, in the code repository, I follow this example to add:
- A
Dockerfileto prepare the Docker container with the software build, and start the application. - A
buildspec-docker.ymlfile with the commands to create and upload the Docker image.
The final workflow is automating all these steps:
- Building the software from the source code.
- Creating the Docker image.
- Uploading of the Docker image to ECR.
- Starting the QA environment on Amazon ECS and Fargate.
- Sending an SNS notification that the QA environment is ready.
The workflow and its steps can easily be customized based on your requirements. For example, with a few changes, you can adapt the buildspec file to push the image to Docker Hub.
Available Now
The CodeBuild service integration is available in all commercial and GovCloud regions where Step Functions and CodeBuild services are offered. For regional availability, please see the AWS Region Table. For more information, please look at the documentation.
As AWS Serverless Hero Gojko Adzic pointed out on the AWS DevOps Blog, CodeBuild can also be used to execute administrative tasks. The integration with Step Functions opens a whole set of new possibilities.
Let me know what are you going to use this new service integration for!
— Danilo