Infrastructure & Automation

A deep dive into testing with TaskCat

In the Up your AWS CloudFormation testing game using TaskCat blog post, I talked about how you can use TaskCat to automate and expedite the testing of your Amazon Web Services (AWS) CloudFormation templates. In this post, I will dive deep into the TaskCat features, various options available, and some tips and tricks for effectively using it to test templates.

taskcat logo
link to taskcat

For this post, I assume that you already have TaskCat installed in your development environment—whether locally on your laptop or in a cloud-based integrated development environment (IDE) like AWS Cloud9. If you don’t have TaskCat installed yet, you can refer to the TaskCat documentation or the Up your AWS CloudFormation testing game using TaskCat blog post.

To get the best results when testing your templates with TaskCat, it’s helpful to understand three primary files that TaskCat uses.

  • Parameter files
    • Standard file(Required)
    • Override files (Optional, but recommended for certain scenarios)
  • Configuration file (Required)

Let’s take a look at each of these files in more detail, and at how you can use them to create and configure the test cases for your CloudFormation templates.

Parameter files

The parameter file is a JSON file that specifies test values for CloudFormation parameters that require user input. When TaskCat launches CloudFormation templates for testing, it uses this parameter file to read the parameter values and pass them to the CloudFormation template for stack creation.

You can have multiple parameter files to test different deployment scenarios for your stack. These parameter files can live anywhere in your project folder. However, we recommend keeping them in the ci folder within your project folder.

Structure

The parameter file must include a ParameterKey and a ParameterValue pair for each parameter in your CloudFormation template that doesn’t have a default value.

[{
    "ParameterKey": "KeyPairName",
    "ParameterValue": "mykey"
}, {
    "ParameterKey": "InstanceType",
    "ParameterValue": "t2.small"
}]

You can also provide a ParameterKey and ParameterValue for a parameter if you want to override the default value that’s in your CloudFormation template.

Tokens

Your CloudFormation template might have parameters that require values to be automatically generated—for example, an alpha-numeric password, user name, etc. TaskCat has pre-defined tokens, which can be used as the ParameterValue in the parameter file. These tokens needs to be specified as $[TOKEN-NAME]. During runtime, TaskCat will generate the parameter value based on the specified token and pass it as a parameter value to the CloudFormation template. Here are the currently supported tokens that generate values in TaskCat.

  • taskcat_genpass_N – Generates a random N character alpha-numeric string. N in the token name specifies the length of the string. For example, $[taskcat_genpass_8] generates tI8zN3iX8.
  • taskcat_genpass_NS – Generates a random N character alpha-numeric string with special characters. N in the token name specifies the length of the string and S specifies that the generated string should also contain special characters. For example, $[taskcat_genpass_8S] generates mA5@cB5!.
  • taskcat_genaz_N – Generates comma-separated values of N Availability Zones. Availability Zone values are generated based on the AWS Region where the stack is launched. N in the token name specifies the number of Availability Zone values to be generated. For example, if the Region is us-east-1, $[taskcat_genaz_2] generates us-east-1a, us-east-2b.
  • taskcat_autobucket – Generates a valid Amazon Simple Storage Service (Amazon S3) bucket name. For example, taskcat-tag-sample-taskcat-project-5fba6597.
  • taskcat_genuuid – Generates a Universally Unique Identifier (UUID). For example, 1c2e3483-2c99-45bb-801d-8af68a3b907b.
  • taskcat_random-string – Generates a random string of length 20 with alphabetic characters only. For example, yysuawpwubvotiqgwjcu.
  • taskcat_random-numbers – Generates a random string of length 20 with numbers only. For example, 56188163597280820763.

In certain use cases, you may want to generate a random string for a parameter and then re-use that same automatically generated value for another parameter of the template. An example is a parameter for the Database password and another parameter to confirm the same Database password. You can generate a random alphanumeric password by using the taskcat_genpass_N token and then refer its value by using another token called $[taskcat_getval_<ParameterKey>]. The following example shows how an alphanumeric value of length 8 is generated for the MyPassword parameter, and then its value is used for the ConfirmMyPassword parameter.

[{
    "ParameterKey": "MyPassword",
    "ParameterValue": "$[taskcat_genpass_8]"
}, {
    "ParameterKey": "ConfirmMyPassword",
    "ParameterValue": "$[taskcat_getval_MyPassword]"
}]

The following image shows an example parameter file (on the left) for a stack that requires user input for seven parameters, and the automatically generated values (on the right) after runtime injection.

parameters that require user input and the values that are generated

Override parameters

In certain situations, you might want to override the defined parameter values with your own values. For example, you might want to override parameters such as KeyPairName or LicenseKey for your own environment. TaskCat supports this functionality with two files:

  • Global override file – Place this in the .aws directory within your home directory: ~/.aws/taskcat_global_override.json
  • Per-project override file – Place this in the ci subfolder of your GitHub repository: <project_name>/ci/taskcat_project_override.json

TaskCat will read parameters in the following order:

  1. Read values from the standard parameter file.
  2. Replace those with values from the global override file, if that file exists.
  3. Replace those with values from the per-project override file, if that file exists.

Configuration file

Another important file required by TaskCat is the configuration file, which is called taskcat.yml in my project. You can name it whatever you want, but you must have this file in your project folder. Though it can be anywhere in your project folder, I recommend putting the configuration file in the ci folder within your project.

Here is the content of TaskCat configuration file for a sample project.

global:
  owner: quickstart@amazon.com
  qsname: test-project
  regions:
    - ap-northeast-1
    - ap-northeast-2
    - ap-south-1
    - ap-southeast-1
    - ap-southeast-2
    - ca-central-1
    - eu-central-1
    - eu-west-1
    - eu-west-2
    - sa-east-1
    - us-east-1
    - us-east-2
    - us-west-1
    - us-west-2
  reporting: true
tests:
  test-scenario1:
    parameter_input: input.json
    template_file: master.template
    regions:
    - us-east-1
  test-scenario2:
    parameter_input: input2.json
    template_file: master.template

The configuration file contains two high-level mappings: global and tests.

The global mapping defines the global configurations of the project. The global configuration contains:

  • owner – The project owner’s email address
  • qsname – The name of the project; it must be same as the project folder name
  • regions – All the AWS Regions where tests need to be executed
  • reporting – Whether to generate a test report with logs from each test execution

The tests mapping defines test scenarios that will be performed by TaskCat. You can define multiple test scenarios in the tests mapping, and each test scenario must specify a parameter input file name and a CloudFormation template file name.

It’s important to note that you only specify the name of the parameter and template files. TaskCat expects to find parameter files and CloudFormation templates in the ci and templates folders, respectively, within the project folder. You can also define the AWS Regions in which the test needs to be executed. This is optional. If you specify a Region for a test scenario, this Region list will override the global Region list, and the test will be performed only in the Regions defined for that test scenario.

Packaging AWS Lambda functions

If you are creating a Lambda function in your CloudFormation template and have its source code as part of your source control repository, there is a good news for you. TaskCat can automatically zip your Lambda function source and upload it to a temporary S3 bucket before launching your CloudFormation templates for testing.

To use this TaskCat feature, you need to put your Lambda function source in the functions/source/<function name>/ folder of your code repository, and add package-lambda: true to the global section of the TaskCat configuration file.

Now, when you run the TaskCat function, folders are zipped and placed in the following location: functions/packages/<Function name>/lambda.zip.

Command line options

When using TaskCat to test your CloudFormation templates, you can use variety of command-line options to enable or disable different features. These options help you further customize your test execution. You can see all the available options by running the taskcat command:

taskcat available options

However, I want to highlight few of the options that can come in really handy when you are testing your CloudFormation templates.

Authentication

By default, TaskCat uses the default AWS credentials configured by you for AWS Command Line Interface (AWS CLI) or boto3. However, you have two ways to override the default and use different credentials when running your tests. This is useful when you work with more than one AWS account and have multiple profiles configured in your AWS credentials file.

First, you can explicitly use a boto profile when running TaskCat. Use the -P option and pass a boto profile name to use it for your test, as in this example.

$ taskcat -c ci/taskcat.yml -P dev

Second, you can use AWS access and secret keys when running TaskCat. Use the -A and -S options to pass the AWS access key and AWS secret key, respectively, when running TaskCat, as in this example.

$ taskcat -c ci/taskcat.yml -A AXXDEFDFEEFD343D -S ADFSFDFD2343SFDSFSDFSS234SFSD

Exclude files and directories

When you run TaskCat, it syncs the content of your source repository to an S3 bucket (either a temporary bucket created by TaskCat or a bucket specified by you in the configuration file), and launches the stack for the templates specified by you from the templates/key prefix within that S3 bucket.

However, in that same source repository, you might have other assets that are not needed by the CloudFormation template, such as license files, copyright notices, static images, readme files, etc. You can exclude those assets from uploading to the S3 bucket by using the -e option. You can exclude whole directories or specific files. You can also use multiple exclude options in a single command. For example, the following command will exclude the images directory and all text files.

$ taskcat -c ci/taskcat.yml -e images -e *.txt

TROUBLESHOOT FAILED stack DURING testing

When you run TaskCat to test your CloudFormation templates, TaskCat launches the stacks according to your test configurations and then deletes them, whether the stack creation completed successfully or failed. However, if you want to keep the stack, for debugging or post deployment verification, either after it has created successfully or failed, TaskCat gives you two options.

Option -n retains the stack irrespective of whether the stack creation was successful or failed.

$ taskcat -c ci/taskcat.yml -n

Option -N retains the stack only if the stack creation failed. On successful creation, the stack will be deleted.

$ taskcat -c ci/taskcat.yml -N

Conclusion

In this post, we covered different parameter files, configuration files, and command line options available in TaskCat. We started with the parameter file and how you can override those parameters using parameter override files. Then I showed how you can configure your tests in the TaskCat configuration file. Following that, we covered how to package Lambda functions that you are deploying as part of your CloudFormation template. Finally, we covered a few key command line options that you can use to customize the behavior of Taskcat when running your tests.

I hope you enjoyed this post and learned some advance features of TaskCat. As always, let us know your feedback in the comments below.