Create a secure and fast DevSecOps pipeline with CircleCI

Historically, public sector developers relied on cumbersome legacy platforms that were costly to maintain and required too much overhead to support high-level software development security. Now, using cloud technology and solutions grounded in development, security, and operations (DevSecOps) practices, the government’s developer community can access the same tools that have long been available to the private sector.

DevSecOps combines the cultural philosophies, practices, and tools of development and operations (DevOps), a methodology which integrates software development with information technology operations, with added security components. With DevSecOps, organizations can deliver secure and compliant application changes rapidly while running operations consistently with automation. In addition, government agencies can be confident in deploying security and privacy controls that meet the federal government’s strict information security requirements.

DevSecOps provides faster build and test cycles, reduced developer wait times, significant savings of management time, as well as greater transparency and control throughout the entire software development lifecycle. Government agencies that transition to DevSecOps can improve the software supply chain security from the initial design through the build, test, deploy, and delivery phases.

In this blog post, we explain how government agencies can accelerate their development workflows while maintaining strict application and operational security using the principles of continuous integration and continuous delivery (CI/CD) and DevSecOps. We provide a solution to walk you through how you can quickly set up your own DevSecOps pipeline that incorporates Amazon Web Services (AWS) and third-party security tools to give you a fast, flexible, and secure software delivery process.

CI/CD and DevSecOps

One way federal agencies can implement DevSecOps is through the continuous integration and continuous delivery (CI/CD) pipeline. The CI/CD pipeline is an automated set of software development processes, from build to test to deploy and release, that help developers deliver value to their users quickly and with increased confidence.

Organizations that adopt CI/CD deliver at high velocity, with 80% of all workflows finishing in less than 10 minutes. In fact, according to Puppet’s 2016 State of DevOps Report, high-performing DevOps teams continue to outperform their organizational counterparts, with 200 times more frequent deployments, 24 times faster recovery from failure, three times lower change failure rate, and 2,555 times shorter lead time. Figure 1 demonstrates where CI/CD fits in the software development pipeline.Figure 1. The software development pipeline. CI/CD fits in the pipeline between the “Store code” and “Deploy” phases.

AWS Partner CircleCI empowers developers with CI/CD to build, test, deploy, and release software with speed, security, and confidence. CircleCI is a shared CI/CD platform with two million daily pipelines running on AWS alone. CircleCI provides CI/CD services for more than one million active developers worldwide at more than 40,000 companies. CircleCI integrates with your version control system (VCS) and automatically builds and tests your application on every commit, notifying your team when pipelines fail and deploying passing applications to your specified staging or production environments (see Figure 2).

Figure 2. CircleCI integrates with your version control system and automatically builds and tests your application on every commit, notifying your team when pipelines fail and deploying passing applications to your specified staging or production environments.

Government agencies use CircleCI for security and DevSecOps

As a CI/CD tool with AWS GovCloud (US) support, SOC-II Type 2 certification, and FedRAMP authorization, CircleCI provides the security that federal agencies require. In addition, CircleCI’s self-hosted solution (i.e. server) runs behind the firewall in AWS GovCloud (US). Government agencies can use AWS GovCloud (US) to comply with federal regulations such as FedRAMP, International Traffic in Arms Regulations (ITAR), Controlled Unclassified Information (CUI) and For Official Use Only (FOUO) data. Government agencies rely on CircleCI to increase developer productivity and deliver modern, scalable solutions to their users while maintaining strict security and compliance standards.

Third-party integrations and DevSecOps

A simple and efficient solution for streamlining and securing integrations across the entire pipeline is an automation tool developed by CircleCI called orbs. Orbs are reusable, shareable, open-source packages of CircleCI config. With just a few lines of code, developers can use orbs to automate repeated processes, speed up project setup, and more simply integrate with third-party tools and services.

Federal developers can access a wide selection of orbs to automate development use-cases such as code analysis, security, testing, and deployment. Some specific examples of CircleCI orbs for automating public sector DevOps include multiple security use cases for vulnerability scanning and secrets management.

Government agencies that want to deploy applications to secure AWS infrastructure or integrate AWS tools into their CircleCI pipelines can start by utilizing orbs for Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), AWS Serverless Application Model (AWS SAM), Amazon Simple Storage Service (Amazon S3), AWS Elastic Beanstalk, AWS CodeDeploy, and AWS Systems Manager Parameter Store.

Figure 3. CircleCI orchestrates development workflows according to the steps specified in your config.yml configuration file. Users can quickly and easily integrate AWS tools into their CI/CD pipeline with just a few lines of code by adding AWS Partner orbs to their configuration.

Implementing a DevSecOps pipeline with CircleCI

Continuous delivery is mission critical in modern day software development, and securing the applications it produces is just as important. DevSecOps-related jobs can be simply defined and executed within CI/CD pipelines on CircleCI using third-party partner tools and orbs to provide solid application vulnerability testing and dynamic application security testing (DAST).

Figure 4. CircleCI can integrate a broad range of AWS services and third-party security tools into all stages of your build, test, and deploy workflow, making it simple to set up a secure and fast DevSecOps pipeline using your preferred solutions.

Integrating security-related jobs in pipelines enables teams to flag and fix security issues as changes are validated. This also empowers developers and security teams to better collaborate around mitigation at the earliest stage of development when security issues are surfaced in the pipeline.

Solution overview

Now, we walk through how to create a CI/CD pipeline with DevSecOps jobs that perform security scans on code changes using CircleCI. Along with AWS services including Amazon S3, Amazon ECR, and Amazon ECS, the pipeline uses additional orbs to integrate security and infrastructure provisioning tools from CircleCI technology partners Snyk, StackHawk, and Terraform.

For every new commit to the main branch of the project repository, the pipeline automatically runs unit tests with Mocha, the JavaScript test framework, as well as static application security testing (SAST) with Snyk, a developer security platform. The pipeline then builds a container image and perform a vulnerability scan on the container environment before pushing the image to Amazon Elastic Container Registry (Amazon ECR). Finally, the pipeline uses StackHawk, an application security testing platform, to start a container instance and perform a DAST scan before deploying the image to an Amazon ECS cluster provisioned by Terraform, an infrastructure-as-code tool to create, update, and version your AWS infrastructure.

Prerequisites

Before you can get started with this tutorial, complete the following tasks:

Fork this example code repo.
Sign up for a CircleCI account
- Set up the aws-devsecops-demo project in CircleCI.
Create an AWS account.
- Create an AWS Identity and Access Management (IAM) user with programmatic access.
  - Assign this user with Amazon ECS permissions.
  - Generate AWS access keys and secrets and save the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for later use.
  - Create an Amazon Elastic Compute Cloud (Amazon EC2) key pair.
- Create an Amazon S3 bucket named aws-devsecops-demo.
  - Enable bucket versioning.
- Create a private Amazon ECR repository named aws-devsecops-demo.
  - Copy the Amazon ECR URL for the repo you just created.
Create a Snyk account.
- Create a Snyk token.
Create a Stackhawk account.
- Create a StackHawk API key.
- Configure your environment and app in StackHawk and record your applicationID.
Create the following project environment variables in CircleCI:
- AWS_ACCESS_KEY_ID: Enter the value of the AWS access key ID.
- AWS_SECRET_ACCESS_KEY: Enter the value of the AWS secret access key.
- AWS_REGION: Enter the value us-east-1.
- AWS_ECR_ACCOUNT_URL: Enter the hostname of your AWS ECR URL (i.e. dkr.ecr.us-east-1.amazonaws.com)
- AWS_ECR_REPO: Enter the name of your ECR repo, aws-devsecops-demo.
- SNYK_TOKEN: Enter the value of the Snyk API token.
- HAWK_API_KEY: Enter the value of the StackHawk API key.
Change values in these files to match your respective values:
- .circleci/config.yml file:
  - Replace the <Specify your AWS Key Pair here> value with your AWS key pair value.
  - Replace the <Add your StackHawk App ID here> value with your StackHawk App ID value.
- terraform/ecs/variables.tf file:
  - Replace the <Add Your AWS Key Pair Here> value with your AWS key pair value.

Implementing DevSecOps jobs within the config.yml file

Pipelines in CircleCI are defined in a config.yml configuration file stored in your VCS. The file identifies the series of jobs to be executed automatically whenever changes are committed and pushed upstream. The config.yml file included in the .circleci directory of the sample repository specifies the DevSecOps-related jobs that provide vulnerability and DAST scans in the example pipeline. It also includes deployment jobs that use Terraform to provision and manage AWS resources leveraged to test the code changes in the target environment. In this demonstration, we create a new Amazon ECS cluster to host the application changes.

In the sections below, we take a closer look at the config.yml file provided in the sample repository to demonstrate how you can define jobs and workflows in your CircleCI DevSecOps pipelines.

CircleCI orbs

Orbs provide quick and easy-to-implement functionality within pipelines. The provided sample project uses a number of different orbs in the config.yml file, including the AWS ECR orb to build and push images to the Amazon ECR, the Snyk orb for vulnerability scans, and the StackHawk orb for dynamic application security testing (DAST), as well as orbs for Node and Terraform.

orbs:
  aws-ecr: circleci/aws-ecr@7.2.0
  snyk: snyk/snyk@0.1.0
  stackhawk: stackhawk/stackhawk@1.0.3
  node: circleci/node@4.2.0
  terraform: circleci/terraform@3.0.0

In the above snippet from the example configuration, the orbs: key introduces the list of orbs this pipeline uses.

Application vulnerability scanning

The Snyk orb provides vulnerability scanning functionality to detect and flag security vulnerabilities in application files.

scan_app:
    docker:
      - image: cimg/node:14.16.0
    steps:
      - checkout
      - run:
          name: Snyk Scan Application files 
          command: npm install 
      - snyk/scan:
          fail-on-issues: false
          monitor-on-build: false

The snippet above defines a job that executes the Snyk orb snyk/scan command, which triggers a dependency vulnerability scan. The scan identifies security vulnerabilities in project dependencies and their severity along with potential mitigation actions. In this example, the fail-on-issues: flag is set to false, which does not fail the build if vulnerabilities are detected. If set to true, the build fails if vulnerabilities are detected. For the purposes of this post, it’s intentionally set to false so that the pipeline reports any issues it discovers but still continue to the next job. In real-world scenarios with high security requirements, this parameter should be set to true.

Container image vulnerability scanning

Snyk’s orb also provides vulnerability scanning functionality that scans container images for vulnerabilities and provides potential mitigation actions for any security weaknesses identified. This scan provides valuable information about the container environment that the application executes in and provides environmental telemetry versus application-related telemetry.

scan_build_docker_image:
    machine:
      image: ubuntu-2004:202101-01
    resource_class: medium
    steps:
      - checkout  
      - docker/check
      - docker/build:
          image: $DOCKER_LOGIN/$CIRCLE_PROJECT_REPONAME
          tag: 0.1.<< pipeline.number >>
      - snyk/scan:
          docker-image-name: $DOCKER_LOGIN/$CIRCLE_PROJECT_REPONAME:0.1.<< pipeline.number >>
          fail-on-issues: false
          monitor-on-build: false
          severity-threshold: high
          target-file: Dockerfile
          token-variable: SNYK_TOKEN
      - docker/push:
          image: $DOCKER_LOGIN/$CIRCLE_PROJECT_REPONAME
          tag: 0.1.<< pipeline.number >>

The above snippet demonstrates how to specify a job that leverages the Snyk orb to perform a vulnerability scan on the container image for this specific build. This container image could be deployed to a production environment, and scanning it for issues provides another important security layer that dramatically reduces potential attack vectors.

The snyk/scan: key again specifies the use of the Snyk orb. The docker-image-name: key and the target-file: key specifications direct the orb to perform a container image scan. The job also leverages the aws-ecr orb, which is used to build and push a container image for this pipeline. The aws-ecr/build-image: and aws-ecr/push-image: commands perform a Docker image build and push that image to the Amazon ECR repository you created.

Performing DAST in CircleCI

Dynamic application security testing (DAST) is the process of scanning an application to find vulnerabilities through simulated attacks. This approach evaluates the app and identifies security vulnerabilities by attacking like a malicious user would.

- stackhawk/hawkscan-local:
          requires:
            - scan_build_docker_image
          docker-network: host
          app-id: bdf5d6af-4c4f-43d9-ba1b-ee305873cb6c
          steps:
            - checkout
            - run:
                name: Run Stackhawk Scan
                command: |
                  docker run -d --rm -p5000:5000 $DOCKER_LOGIN/$CIRCLE_PROJECT_REPONAME:0.1.<< pipeline.number >>

The above code snippet demonstrates how to specify a job that performs a DAST scan on the application changes in this build. This particular job is based on the StackHawk orb, which performs the stackhawk/hawkscan-local: job. The hawkscan-local: call in the orb is a predefined job and must be triggered within the pipeline’s workflow: section. Workflows orchestrate pipeline jobs, hence the reason this must be specified as a job to trigger in the workflow.

The app-id: specifies the Stackhawk app ID associated with this repository. The ID for the sample application has been included in the configuration file provided in example. If you are setting up your own pipeline for a different application, you need to update this parameter with your own app ID.

The Run Stackhawk Scan command starts a container instance based on the image built in previous jobs, then the StackHawk orb performs a DAST scan on that container.

The DAST configuration is specified in the stackhawk.yml file, which must also have the app-id specified in it. Replace the < Add your App ID here> syntax with your app-id value for the orb to associate the test with your application. The DAST results can be seen in the CircleCI dashboard, but a more robust view of these results can be viewed from the StackHawk application portal.

Deploying to Amazon ECS using Terraform

The last piece of this pipeline is the deploy job that provisions an Amazon ECS cluster and deploys the build changes in the form of a container. This job deploys the container to a target environment and verifies that the deployment functions on its intended target. This also enables more robust testing of the changes in the target environment. Testing in this environment provides insight into the unknown and provide a solid test bed for the changes in a safe environment.

terraform_deploy_aws_ecs:
    machine:
      image: ubuntu-2004:202101-01
    resource_class: medium
    steps:
      - checkout
      - run:
          name: Create .terraformrc file locally
          command: echo "credentials \"app.terraform.io\" {token = \"$TERRAFORM_TOKEN\"}" > $HOME/.terraformrc
      - terraform/install:
          terraform_version: "1.0.2"
          arch: "amd64"
          os: "linux"
      - terraform/init:
          path: ./terraform/ecs
      - terraform/plan:
          path: ./terraform/ecs
      - run:
          name: Terraform apply
          command: |
            terraform -chdir=./terraform/ecs apply \
              -var docker_img_name=${DOCKER_LOGIN}/${CIRCLE_PROJECT_REPONAME} \
              -var docker_img_tag=0.1.<< pipeline.number >> \
              -var key_pair=devrel-angel-rivera \
              -auto-approve
            export ENDPOINT="$(terraform -chdir=./terraform/ecs output load_balancer_hostname)"
            mkdir -p /tmp/ecs/
            echo 'export ENDPOINT='${ENDPOINT} > /tmp/ecs/endpoint
      - persist_to_workspace:
          root: /tmp/ecs/
          paths:
            - "*"      
      - run: sleep 90

This code snippet demonstrates how to leverage the Terraform orb to provision an Amazon ECS cluster and deploy the container image changes. This pipeline job uses a machine executor along with the CircleCI Ubuntu image for pipeline runtime. The Amazon ECS cluster is powered by AWS Graviton EC2 nodes and the Amazon Linux 2 Amazon Machine Image (AMI).

The Terraform code for this project lives in the terraform/ecs directory, which defines the Amazon ECS cluster and associated AWS Cloud resources. This job installs the Terraform client, initializes the Terraform project, and finally executes the provisioning process with the terraform -chdir=./terraform/ecs apply command. The -var parameters specified in the apply command specify values that are used in the Terraform process. After successful execution of this job, a new Amazon ECS cluster is created and can be used to perform extensive testing.

Learn more about DevSecOps and AWS for public sector

In this tutorial, you learned how to configure a secure DevSecOps pipeline to automatically build, test, and deploy a containerized application to an Amazon ECS cluster. DevSecOps is becoming a new benchmark for the public and private sector thanks to its efficiency, effectiveness, visibility, and cost savings. Government agencies that transition to DevSecOps can improve the software supply chain security and overall developer agility from the initial design through the build, test, deploy, and delivery phases.

You can find more DevSecOps tools and resources in the AWS marketplace. To learn more about how you can implement DevSecOps pipelines using AWS services and third-party tools, read the blog posts, “Building end-to-end AWS DevSecOps CI/CD pipeline with open source SCA, SAST and DAST tools” and “Building an end-to-end Kubernetes-based DevSecOps software factory on AWS.”

Agencies with longer procurement processes can take advantage of a no cost trial or private offer of CircleCI in the AWS Marketplace. To get started, sign up for CircleCI or contact CircleCI for more information.

AWS Public Sector Blog