Automating your API testing with AWS CodeBuild, AWS CodePipeline, and Postman

Today, enterprises of all shapes and sizes are engaged in some form of digital transformation. Many recognize that successful digital transformation requires continuous evolution powered by a robust API strategy. APIs enable the creation of new products, improvement of the customer experience, transformation of business processes, and ultimately, the agility needed to create sustainable business value. Hence, it stands to reason that adopting DevOps best practices such as Continuous Integration into your API development lifecycle helps improve the quality of your APIs and accelerate your API strategy.

In this post, we highlight how to automate API testing using serverless technologies, including AWS CodePipeline and AWS CodeBuild, along with Postman. AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages ready to deploy without the need to provision, manage, and scale your own build servers.

We take advantage of a new feature in CodeBuild called Reports that allows us to view the reports generated by functional or integration tests. We keep an eye on valuable metrics such as Pass Rate %, Test Run Duration, and the number of Passed versus Failed/Error test cases.

Postman is an industry-recognized tool used for API development that makes it easy to both develop and test your APIs. Postman also includes command-line integration with its command-line Collection Runner, Newman. Newman can easily be integrated with your continuous integration servers and build systems. Our CodePipeline pipeline uses CodeBuild to invoke the Newman command line interface and execute tests created with Postman. We cover the steps in detail below.

Solution Overview

In this post, we demonstrate how to automate the deployment and testing of the Pet Store API that is available as a sample API with API Gateway. This is a simple API that integrates via HTTP proxy to a demo Pet Store API. The API contains endpoints to list pets, get a pet by specific id, and add a pet.

The following diagram depicts the architecture of this simple Pet Store demo API.

The following diagram depicts the AWS CodePipeline pipeline architecture we use to test the PetStore API.

After execution of this pipeline, you have a fully operational API that has been tested for specific functional requirements. These test cases and their results are visible in the Reports section of the CodeBuild console.

Building the PetStore API Pipeline

To get started, follow these steps:

Step 1. Fork the Github repository

Log into your GitHub account and fork the following repository: https://github.com/aws-samples/aws-codepipeline-codebuild-with-postman

Step 2. Clone the forked repository

Clone the forked repository into your local development environment.
git clone https://github.com/<YOUR_GITHUB_USERNAME>/aws-codepipeline-codebuild-with-postman

Step 3. Create an Amazon S3 bucket

This bucket contains resources related to this project. We refer to this bucket as the project’s root bucket.
Using the AWS CLI: aws s3 mb s3://<REPLACE_ME_WITH_UNIQUE_BUCKET_NAME>

Step 4. Edit the buildspec file

The buildspec file petstore-api-buildspec.yml contains the instructions to package the resources defined in your SAM template, petstore-api.yaml. This build spec is executed by CodeBuild within the build stage (BuildPetStoreAPI) of the pipeline.

1. Replace the following text REPLACE_ME_WITH_UNIQUE_BUCKET_NAME in the petstore-api-buildspec.yml with the bucket name created above in step 3.

2. Commit this change to your repository.

Step 5. Store Postman collection and environment files in S3

1. Navigate to the directory 02postman

For this project we included a Postman collection file, PetStoreAPI.postman_collection.json, that validates the PetStore API’s functionality. You can import the collection and environment file into Postman using the instructions here to see the tests associated with each API endpoint.

The following screenshot is an example specific to testing a GET request to the /pets endpoint(1). We make sure the GET request returns a JSON array(2) along with the inclusion of a Content-Type header (3) and a response time of less than 200ms (4). In the Test results tab (5), you can see we passed these tests when calling the API.

2. Save the Postman collection file in S3 using the AWS CLI

aws s3 cp PetStoreAPI.postman_collection.json \
s3://<REPLACE_ME_WITH_UNIQUE_BUCKET_NAME>/postman-env-files/PetStoreAPI.postman_collection.json

3. Save the Postman environment file to S3 using the AWS CLI

aws s3 cp PetStoreAPIEnvironment.postman_environment.json \
s3://<REPLACE_ME_WITH_UNIQUE_BUCKET_NAME>/postman-env-files/PetStoreAPIEnvironment.postman_environment.json

Step 6. Create the PetStore API pipeline

We now create the AWS CodePipeline PetStoreAPI pipeline that will both deploy and test our API. We use AWS CloudFormation template (petstore-api-pipeline.yaml) to define the pipeline and required stages, as noted in our pipeline architecture diagram.

Navigate back to the project’s root directory

To launch this template, you need to fill in a few parameters:
BucketRoot: unique bucket folder you created above
GitHubBranch: master
GitHubRepositoryName: aws-codepipeline-codebuild-with-postman
GitHubToken: your github personal access token
You can create your github token here (for select scopes: check repo and admin:repohook)
GitHubUser = your github username

2. Use the AWS CLI to deploy the AWS CloudFormation template as follows

aws cloudformation create-stack --stack-name petstore-api-pipeline \
--template-body file://./petstore-api-pipeline.yaml \
--parameters \
ParameterKey=BucketRoot,ParameterValue=<REPLACE_ME_WITH_UNIQUE_BUCKET_NAME> \
ParameterKey=GitHubBranch,ParameterValue=<REPLACE_ME_GITHUB_BRANCH> \
ParameterKey=GitHubRepositoryName,ParameterValue=<REPLACE_ME_GITHUB_REPO> \
ParameterKey=GitHubToken,ParameterValue=<REPLACE_ME_GITHUB_TOKEN> \
ParameterKey=GitHubUser,ParameterValue=<REPLACE_ME_GITHUB_USERNAME> \
--capabilities CAPABILITY_NAMED_IAM

This command creates a CodePipeline pipeline and required stages to deploy and test our API using CodeBuild and Newman. Open the CodePipeline console to watch your pipeline execute and monitor the different stages, as shown in the following screenshot.

The last stage of the pipeline uses CodeBuild and Newman to execute the tests created with Postman. You should now have a fully functional API visible in the Amazon API Gateway console.

Review AWS CodeBuild configuration

For this pipeline, we use CodeBuild to both deploy our API in the build stage and to test our API in the test stage of the pipeline. For the deploy stage, CodeBuild uses AWS Serverless Application Model (SAM) to build and deploy our API. We focus on the test stage and how we use CodeBuild to run functional tests against our API.

Take a look at the buildspec file (postman-newman-buildspec.yml)that CodeBuild uses to execute the test. Recall that our goal for this stage is to execute functional tests that we created earlier using Postman and to visualize these test results in CodeBuild Reports:

version: 0.2

env:
  variables:
    key: "S3_BUCKET"

phases:
  install:
    runtime-versions:
      nodejs: 10
    commands:
      - npm install -g newman
      - yum install -y jq

  pre_build:
    commands:
      - aws s3 cp "s3://${S3_BUCKET}/postman-env-files/PetStoreAPIEnvironment.postman_environment.json" ./02postman/
      - aws s3 cp "s3://${S3_BUCKET}/postman-env-files/PetStoreAPI.postman_collection.json" ./02postman/
      - cd ./02postman
      - ./update-postman-env-file.sh

  build:
    commands:
      - echo Build started on `date` from dir `pwd`
      - newman run PetStoreAPI.postman_collection.json --environment PetStoreAPIEnvironment.postman_environment.json -r junit

reports:
  JUnitReports: # CodeBuild will create a report group called "SurefireReports".
    files: #Store all of the files
      - '**/*'
    base-directory: '02postman/newman' # Location of the reports

In the install phase, we install the required Newman library. Recall this is the library that uses Postman collection and environment files to execute tests from the CLI. We also install the jq library that allows you to query JSON.

In the pre_build phase, we execute commands that set up our test environment. In this case, we need to grab the Postman collection and environment files from Amazon S3. Then we use a shell script, update-postman-env-file.sh to update the Postman environment file with the API Gateway URL for the API created in the build stage. Lets take a look at the shell script executed by CodeBuild:

#!/usr/bin/env bash

#This shell script updates Postman environment file with the API Gateway URL created
# via the api gateway deployment

echo "Running update-postman-env-file.sh"

api_gateway_url=`aws cloudformation describe-stacks \
  --stack-name petstore-api-stack \
  --query "Stacks[0].Outputs[*].{OutputValueValue:OutputValue}" --output text`

echo "API Gateway URL:" ${api_gateway_url}

jq -e --arg apigwurl "$api_gateway_url" '(.values[] | select(.key=="apigw-root") | .value) = $apigwurl' \
  PetStoreAPIEnvironment.postman_environment.json > PetStoreAPIEnvironment.postman_environment.json.tmp \
  && cp PetStoreAPIEnvironment.postman_environment.json.tmp PetStoreAPIEnvironment.postman_environment.json \
  && rm PetStoreAPIEnvironment.postman_environment.json.tmp

echo "Updated PetStoreAPIEnvironment.postman_environment.json"

cat PetStoreAPIEnvironment.postman_environment.json

This shell script wraps AWS API commands to get the required API Gateway URL from the AWS CloudFormation stack output and uses this value to update the Postman environment file. Notice how we also use the jq library installed earlier.

Once this is done, we move on to the build phase in our postman-newman-buildspec.yml. Note in the commands section how we execute the Newman command line runner by passing the required Postman collection and environment files. Also, notice how we specify to Newman that we want these reports in JUnit style output. This is very important, as this allows CodeBuild Reports to consume and visualize this output.

Once our test run is complete, we specify in our buildspec file where our test results JUnit files are available. This allows CodeBuild Reports to consume our JUnit test results for visualization.

You can accomplish all of this without having to provision, manage, and scale your own build servers.

Working with CodeBuild’s test reporting feature

CodeBuild announced a new reporting feature that allows you to view the reports generated by functional or integration tests. You can use your test reports to view trends and test and failure rates to help you optimize builds. The test file format can be JUnit XML or Cucumber JSON. You can create your test cases with any test framework that can create files in one of those formats (for example, Surefire JUnit plugin, TestNG, or Cucumber).

Using this feature, you can see the history of your test runs and see duration for the entire report, as shown in the following screenshots:

It also provides details for individual test cases within a report, as shown in the following screenshot.

You can select any individual test case to see its details. The following screenshot shows details of a failed test case.

Please note that at the time of this publication, the CodeBuild reporting feature is in preview.

Cleanup

After the tests are completed, we recommend the following steps to clean-up the resources created in this post and avoid any charges.

1. Delete the AWS CloudFormation stack petstore-api-stack to delete the PetStore API deployed by the pipeline stack

2. Delete the pipeline artifact bucket created by the petstore-api-pipeline stack.

This is the bucket referred to as CodePipelineArtifactBucket in the resources tab of the petstore-api-pipeline stack and begins with the name: petstore-api-pipeline-codepipeline-artifact-bucket. This bucket needs to be deleted in order to delete the pipeline stack.

3. Delete the AWS CloudFormation stack petstore-api-pipeline to delete the AWS CodePipeline pipeline that builds and deploys the PetStore API.

Conclusion

Continuous Integration is a DevOps best practice that helps improve software quality. In this blog post, we showed how you can use AWS Services such as CodeBuild and CodePipeline with Postman, a powerful API testing and development tool, to easily adopt Continuous Integration and DevOps best practices into your API development process.