Infrastructure & Automation

Up your AWS CloudFormation testing game using TaskCat

Testing the code to ensure it does what you expect it to do is critical in every application development process. With Infrastructure as code, the same principle applies to the code written to create infrastructure resources.

AWS CloudFormation provides a common language to describe and provision all the infrastructure resources in the AWS Cloud environment. It allows you to use either the JSON or the YAML file format to model and provision AWS resources in an automated and secure manner across all AWS Regions and accounts. With AWS CloudFormation templates, you can model your entire AWS infrastructure in code and apply standard best practices such as code reviews, version control, and testing.

TaskCat is an open source tool developed by the AWS Quick Start team to automate the testing of AWS CloudFormation templates. It tests an AWS CloudFormation template by deploying it in multiple AWS Regions simultaneously and generates a report with a pass/fail result for each Region. You can customize the tests through a test config file. TaskCat is implemented in Python and available as a Docker container and pip module.

In the rest of this blog post, I will walk you step-by-step through how to install and use TaskCat to test AWS CloudFormation templates.

Installing TaskCat

TaskCat can be installed with Docker or by using pip3. (TaskCat on Windows is not supported. If you are running Windows 10 we recommend that you install Windows Subsystem for Linux (WSL) and then install TaskCat inside the WSL environment. For details, see Install and configure TaskCat on Microsoft Windows 10.)

If you are installing by using Docker, you can run following command.

curl -s | sh

Simply pulling the Docker container using the docker pull command won’t work. The script wraps the Docker run command with the options needed so that ~/.aws can be accessed from inside the container.

If you don’t have root privileges, TaskCat will be installed in the directory you are running the command from.

If you have Python 3.5+ installed and you want to install TaskCat by using pip3, run the following command.

pip3 install taskcat --user

It will install TaskCat into your home directory. I recommend using a virtual environment when installing TaskCat using pip3.

The installation directory for TaskCat is platform-specific. On macOS, it’s ~/Library/Python/3.x/bin/taskcat (make sure to add the Python bin directory to your $PATH). On Linux, it’s ~/.local/bin.

I am using pip3 to install TaskCat, and therefore my installation looks like the following:

screenshot of code

Preparing Tests

TaskCat expects a specific directory structure where it looks for the AWS CloudFormation templates and configuration files. Templates are expected to be in the templates/ folder and configuration files in the ci/ folder. To run the tests, TaskCat requires two files in the ci/ folder within the project directory:

  • A parameter file in JSON format
  • A configuration file named taskcat.yml

I’ve created these two files in my project directory. My parameter file is called input.json.

file structure

Make sure that input.json and taskcat.yml are in the ci/ folder in your project directory.

The parameter file contains the test values for the parameters of your AWS CloudFormation template. You can have multiple parameter files to test different deployment scenarios. This file is a list of objects, and each object contains the following two keys:

  • ParameterKey specifies the exact name of the parameter in the AWS CloudFormation template.
  • ParameterValue specifies the parameter value that needs to be passed to the parameter.

You can also autogenerate values for some parameter types, such as Universally Unique IDs (UUIDs), Regions, strings, and numbers, at runtime by using predefined tokens. For the complete list of predefined tokens, see the TaskCat documentation. The following image shows the parameter file I am using to test my AWS CloudFormation template.

parameters listed in code

My parameter file includes a parameter called Password, which I don’t want to check in to GitHub. For such situations, TaskCat provides the capability to override the parameter values via two override files:

  • Global override file – Place this in the .aws directory within your home directory ~/.aws/taskcat_global_override.json.
  • Project override file – Place this in the ci/ directory within your project directory <project_name>/ci/taskcat_project_override.json.

TaskCat will read parameters in the following order:

  • Read values from the standard parameter files
  • Replace those with values from the global override file, if that file exists
  • Replace those with values from the project override file, if that file exists

Next, we will create a TaskCat configuration file named taskcat.yml in the ci/ folder of the project directory. My taskcat.yml file is at ~/test-project/ci/taskcat.yml:

  qsname: test-project
    - ap-northeast-1
    - ap-northeast-2
    - ap-south-1
    - ap-southeast-1
    - ap-southeast-2
    - ca-central-1
    - eu-central-1
    - eu-west-1
    - eu-west-2
    - sa-east-1
    - us-east-1
    - us-east-2
    - us-west-1
    - us-west-2
  reporting: true
    parameter_input: input.json
    template_file: master.template
    - us-east-1
    parameter_input: input2.json
    template_file: master.template

The taskcat.yml file contains two high-level mappings: global and tests. The global mapping defines the global configurations of the project:

  • owner – Project owner’s email address
  • qsname – Name of the project; must be same as the project folder name
  • regions – All the regions where tests need to be executed
  • reporting – To generate test report with logs from each test execution

The tests mapping defines test scenarios that will be performed by TaskCat. You can define multiple test scenarios in the tests mapping, and each test scenario must specify a parameter input file name and an AWS CloudFormation template file name. Optionally, you can define the AWS Regions in which the test needs to be executed. This Region list will override the global Region list. For example, if your AWS CloudFormation template creates AWS resources that are not available in all Regions, you need to override the global Region list and specify only those Regions where those resources are supported.

Running TaskCat

At this point we have AWS CloudFormation templates and TaskCat configurations in place, and we’re ready to run our tests. To start the test, run the following command:

$ taskcat -c <project-directory>/ci/taskcat.yml

In addition to -c, the TaskCat command line interface (CLI) provides several options to add capabilities to your test run, giving you more control over it. You can see all the supported options by just running $ taskcat at your command prompt.

screen output listing all options

To test my AWS CloudFormation templates, I am running the following command:

$ taskcat -c test-project/ci/taskcat.yml

where test-project is my project name and the directory containing project files, and taskcat.yml is the TaskCat configuration file.

TaskCat performs multiple actions, such as template validation, parameter validation, and staging content into an Amazon Simple Storage Service (Amazon S3) bucket, before launching the AWS CloudFormation stack. It launches the stack in the defined Regions, simultaneously. And it regularly polls the AWS CloudFormation stack status to check if the stack creation is finished. After the stack has been successfully created, TaskCat deletes it. How much time TaskCat takes to finish the testing depends on how many tests you have defined in your TaskCat configuration file and how long each stack creation and deletion takes. After the TaskCat run is completed, it generates a report in HTML format in the directory from where you are running the TaskCat command. You can see the report by opening the file named index.html in <current-directory>/taskcat_outputs/. The following image shows the report of my test run.

taskcat report


In this post, we covered how to install TaskCat, configure and prepare tests, and run TaskCat to test AWS CloudFormation templates. TaskCat is a powerful tool that automates and expedites the testing of AWS CloudFormation templates and provides consolidated test report, which makes it easy to read and share with others.

In an upcoming blog post, I will deep dive into TaskCat and go behind the scenes of it. I hope you enjoyed this post. Let us know your feedback in the comments!