Build infrastructure continuous integration for Terraform code leveraging AWS Developer Tools and Terratest

Introduction

Day by day customers are embracing Infrastructure as Code (IaC) as a main approach to design, provide, develop and maintain their cloud infrastructure. IaC ensures reproducibility and scalability of their infrastructure while at the same time it allows DevOps to adopt best practices and paradigms. In the context of IaC, Terraform is widely used among customers to provision and manage infrastructure and services on AWS. Terraform allows developers and infrastructure architects to organize the code in reusable and shareable modules. Modules can, and should, be used to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.

As a consequence performing automated tests on the IaC becomes a key operation. These automated tests ensure quality, stability, reliability and security of the code, hence, of the provisioned resources. Due to the peculiar nature of the Infrastructure as Code, setting up automated test pipelines is challenging and often force the developers to adopt a not standard self-customized solutions.

Solution Overview

The scope of this blog post is to provide a guide and ready to use code to setup a complete Continuous Integration (CI) pipeline with end-to-end tests based on AWS CodePipeline, AWS CodeBuild, AWS CodeCommit, Terraform and Terratest.

As an outcome customers will be not only be able to utilize the complete CI Pipeline but also to extend and customize the code to fit their own needs. The following architecture diagram simplifies the proposed solution. Note that an independent account can be used to run the end-to-end tests based on the provided Terraform modules.

Repository Structure

As a starting point please download the contents of the following GitHub repository
A description of the structure of the code is found below. It will help navigate through the proposed example:

.
|-- README.md
`-- terraform
|-- examples
`-- modules

/terraform

This directory contains:
./examples reusable examples for the modules.
./modules Terraform modules for the CI pipeline, including the CodeBuild build scripts.
The content of the repository is structured providing reusable Terraform modules.
These modules and their content can be extracted and reused.

/terraform/examples

It is a good practice when releasing modules to provide also working examples. Those examples can be stored in a dedicated directory at the same level of the modules directory.

In this repository the example cicd_account (on the location: /terraform/examples/cicd_account/main.tf) is provided. It uses the module located in /terraform/modules/cicd. This example will deliver a CI based on CodePipeline, CodeBuild and CodeCommit and uses this same repository as source.

Note that this proposed behaviour can be changed and adapted to the specific needs.

/terraform/modules

This directory contains the Terraform modules, the tests and the code run by CodeBuild.

Navigate to Build specification reference for CodeBuild for information about CodeBuild and how to interpret the buildspec.yaml file.

In a nutshell the workflow of CodeBuild is:

1. It runs the tests described in modules/cicd/test
2. For each run the test report can be accessed via AWS CodeBuild Console under the Reports.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• An AWS Account (second account optional)
• Terraform version ≥ 12.0
• aws cli v2
• python ≥3.8
• go ≥1.15.2
• Terratest

Walkthrough

1. Deploy CI Pipeline

Using the provided code example the CI pipeline can be deployed. One has the choice of deploying the cross account setup, where the CI runs on one AWS account and the tests are performed on another AWS account. For the sake of simplicity this guide walks you through setting up the solution within a single AWS account.

To deploy the CI pipeline follow the these steps:

1. Modify the file under: terraform/examples/cicd_account/inventories/variables.tfvars with the desired name of your repository:
   git_repository_name="artifact-cicd-test"
2. Configure your AWS CLI
3. Deploy the resources with and follow the instructions provided:
   cd terraform/examples
./run_terraform.sh cicd_account

Here’s what you’ve deployed to your account —

– AWS CodeCommit – Fully-managed source control service that hosts the secure Git-based repository.
– AWS CodeBuild – Managed build service to run continuous integration, hence the terraform tests.
– AWS CodePipeline – fully managed continuous delivery service in charge of automating the build/test stag
– AWS IAM Roles and Policies describing least privilege permissions required for the scope of this example.

You can use this Troubleshooting guide for common errors or misses during the deployment.

2. Review and tune the test code

A test example called cicd_test.go is provided in the location terraform/modules/cicd/test
The example can be extended to meet the desired or required requirements. This step-by-step guide to the module cicd can also be found in the terraform/modules/cicd/README

3. Push the code inside your new CodeCommit repository.

For this sample test you can use the same code you used in Step 1 as if it were the code you are willing to test. Use this guide to push the terraform code to the created CodeCommit repository:

1. This first step is optional, but it is recommended: create a .gitignore file in the root level of the repository to avoid git tracking Terraform state:

echo ".tfstate*
.terraform
plan.out" >> .gitignore

2. Retrieve repository remote address:

1. On the AWS console go in CodeCommit
2. Click on your repository name
3. Click on Clone URL and select your preferred way (it is recommended HTTPS (GRC)).
4. Extended documentation about how to set up it can be found by clicking on connection steps.

3. Setup your git repo:

git remote add origin <THE_URL_YOU_GOT_ON_STEP2>
## By default the CodePipeline monitors the dev branch
git checkout -b dev
git push --set-upstream origin

4. Push the repo against the dev branch this will trigger the start of the Pipeline
5. Follow the pipeline run by connecting to the CodePipeline console.

4. Cleanup

When finished you can easily clean up your account by following the next steps:

cd terraform/examples
./run_terraform.sh cicd_account delete

Troubleshooting

Terraform Error when deleting S3 Bucket (BucketNotEmpty)

Terraform can exit with this error when you are breaking down or updating the infrastructure. The reason resides to the fact that Terraform will not remove a non-empty S3 bucket. In order to fix the issue you should manually remove the object in the S3 Bucket:

1. Access to AWS Console and go to S3
2. Select the targeted bucket and click on Empty
3. Confirm the deletion on the console
4. When the operation is finished try to re-run the terraform command, it should now run successfully.

As an alternative you may also make use of force_destroy argument in the aws_s3_bucket Terraform resource. Follow this link for more information

Conclusion

After following this guide, customers are now able to set up a fully automated CI pipelines for testing Terraform code by using Terratest libraries, AWS CodePipeline, AWS CodeBuild, AWS CodeCommit. By doing so customers are now also familiar with the testing principle for IaC, more specifically for Terraform.

About the Authors