AWS Developer Tools Blog
Testing CDK Applications in Any Language
The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to define your cloud application resources using familiar programming languages. Because the AWS CDK enables you to define your infrastructure in regular programming languages, you can also write automated unit tests for your infrastructure code, just like you do for your application code. Testing is an essential element to highly effective DevOps practices, and testing your infrastructure code provides benefits such as ensuring that you will create exactly the resources you expect in the AWS cloud and helping to prevent regressions from being introduced to your infrastructure.
Today, I am happy to announce the assertions module for the AWS Cloud Development Kit, a set of APIs designed to help you write unit tests against your CDK applications, with a focus on CloudFormation templates.
Cross-Language Support
A previous AWS blog post explains how to write tests for your infrastructure constructs using the assert module, which is available only for JavaScript and TypeScript.
Similar to the assert module, the new CDK assertions module provides a robust set of APIs to precisely verify the CloudFormation templates synthesized by your CDK app. Additionally, the assertions module is available for every language supported by the CDK.
While the new assertions module supports every language that is supported by the CDK, this snippets in this article will be written in Python. However, the full source code for these examples is available on GitHub, and contains equivalent code written in TypeScript, Java, and Python.
If you are currently using the
assertmodule and wish to migrate, you can find a guide on that in our GitHub repository.
Fine-grained Assertions
The assertions module provides several tools to both assert that certain parts of a template matches given objects and to retrieve certain parts of of a template. Using these tools, we can assert that resources with a given type and properties exist, assert that certain outputs exist, and assert that a template has a given number of resources.
Let’s assume you have a stack that creates an AWS Step Functions state machine and an AWS Lambda function. The Lambda function is subscribed to an Amazon SNS topic and simply forwards the message to the state machine:
from typing import List
from aws_cdk import aws_lambda as lambda_
from aws_cdk import aws_sns as sns
from aws_cdk import aws_sns_subscriptions as sns_subscriptions
from aws_cdk import aws_stepfunctions as sfn
from aws_cdk import core as cdk
class ProcessorStack(cdk.Stack):
def __init__(
self,
scope: cdk.Construct,
construct_id: str,
*,
topics: List[sns.Topic],
**kwargs
) -> None:
super().__init__(scope, construct_id, **kwargs)
# In the future this state machine will do some work...
state_machine = sfn.StateMachine(
self, "StateMachine", definition=sfn.Pass(self, "StartState")
)
# This Lambda function starts the state machine.
func = lambda_.Function(
self,
"LambdaFunction",
runtime=lambda_.Runtime.NODEJS_14_X,
handler="handler",
code=lambda_.Code.from_asset("./start-state-machine"),
environment={
"STATE_MACHINE_ARN": state_machine.state_machine_arn,
},
)
state_machine.grant_start_execution(func)
subscription = sns_subscriptions.LambdaSubscription(func)
for topic in topics:
topic.add_subscription(subscription)How do you test this stack with fine-grained assertions? First, start by creating the ProcessorStack and synthesizing it into a CloudFormation template. Once you have synthesized your stack into a Template, you can assert on it. You’ll notice that every test using the assertions module starts with first creating a stack and synthesizing it.
Additionally, since this stack relies on cross-stack references, you can create a stack for the referenced resources to live in and pass those resources to the ProcessorStack:
from aws_cdk import aws_sns as sns
from aws_cdk import core as cdk
from aws_cdk.assertions import Template
from app.processor_stack import ProcessorStack
def test_synthesizes_properly():
app = cdk.App()
# Since the ProcessorStack consumes resources from a separate stack
# (cross-stack references), we create a stack for our SNS topics to live
# in here. These topics can then be passed to the ProcessorStack later,
# creating a cross-stack reference.
topics_stack = cdk.Stack(app, "TopicsStack")
# Create the topic the stack we're testing will reference.
topics = [sns.Topic(topics_stack, "Topic1")]
# Create the ProcessorStack.
processor_stack = ProcessorStack(
app, "ProcessorStack", topics=topics # Cross-stack reference
)
# Prepare the stack for assertions.
template = Template.from_stack(processor_stack)Now you can assert that the Lambda function and a subscription were created:
# ...
# Assert it creates the function with the correct properties...
template.has_resource_properties(
"AWS::Lambda::Function",
{
"Handler": "handler",
"Runtime": "nodejs14.x",
},
)
# Creates the subscription...
template.resource_count_is("AWS::SNS::Subscription", 1)
# ...The has_resource_properties() method allows you to assert that the template has a resource of the given type with the given properties. This example asserts that each of the expected resources exist and they are configured with the specific properties defined in the tests. There are also many other methods on the Template class which can be used to verify the Resources, Outputs, and Mappings sections of the CloudFormation template.
Matchers
Looking at the assertions in the previous example, you will notice that the tested properties are all defined as literal values. However, the methods in the assertions module also accept special matchers which allow you to define partial or special pattern matching during template assertions. There are several matchers built into the assertions module, with the full list documented in the assertions API reference. For example, the Match.object_like() method, checks that the expected values (the properties passed to has_resource_properties()) are a subset of the target values (the properties on the synthesized resource). The Match.object_like() matcher is commonly used to prevent tests from failing when additional (possibly unrelated) properties are introduced by the CDK construct library.
The next examples show you how to use various matchers in your unit tests. The Match.object_equals() matcher checks that the expected values are exactly equal to the target values, rather than a subset. On the other hand, the Match.any_value() matcher allows the assertion to pass no matter what the target value is. Using a combination of these two matchers, you can fully assert on the state machine’s IAM role:
from aws_cdk.assertions import Match
# ...
# Fully assert on the state machine's IAM role with matchers.
template.has_resource_properties(
"AWS::IAM::Role",
Match.object_equals(
{
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": {
"Fn::Join": [
"",
[
"states.",
Match.any_value(),
".amazonaws.com",
],
],
},
},
},
],
},
}
),
)In the next example, assertions for the Step Functions state machine are included in the unit tests. State machines are defined by strings written in the Amazon State Languages (based on JSON). While this is a great way to express state machines, it is difficult to test without any tools to help out. Fortunately, the assertions module provides a matcher, Match.serialized_json(), which JSON-deserializes the target string and matches it against the expected value. You can even nest matchers inside of a Match.serialized_json() matcher!
The following example uses Match.serialized_json() to assert on the state machine’s definition:
# ...
# Assert on the state machine's definition with the serialized_json matcher.
template.has_resource_properties(
"AWS::StepFunctions::StateMachine",
{
"DefinitionString": Match.serialized_json(
# Match.object_equals() is used implicitly, but we use it explicitly
# here for extra clarity.
Match.object_equals(
{
"StartAt": "StartState",
"States": {
"StartState": {
"Type": "Pass",
"End": True,
# Make sure this state doesn't provide a next state --
# we can't provide both Next and set End to true.
"Next": Match.absent(),
},
},
}
)
),
},
)Capturing
The Capture API of the assertions module allows you to retrieve values the assertions module encounters when it is matching and perform your own assertions on those values later. You create a Capture object, use it in your assertion just as if it was any other matcher, and then retrieve its value with the relevant as_x() method (for example, as_string() or as_object()).
This snippet uses a combinations of Captures and the Match.serialized_json() matcher to assert that the name of the start state of a state machine starts with “Start” and that the start state is actually present within the list of states in the machine, doing some more rudimentary validation on the state machine definition:
import re
from aws_cdk.assertions import Capture
# ...
# Capture some data from the state machine's definition.
start_at_capture = Capture()
states_capture = Capture()
template.has_resource_properties(
"AWS::StepFunctions::StateMachine",
{
"DefinitionString": Match.serialized_json(
Match.object_like(
{
"StartAt": start_at_capture,
"States": states_capture,
}
)
),
},
)
# Assert that the start state starts with "Start".
assert re.match("^Start", start_at_capture.as_string())
# Assert that the start state actually exists in the states object of the
# state machine definition.
assert start_at_capture.as_string() in states_capture.as_object()Snapshot Testing
Another way to test your CDK applications is using snapshot tests. Snapshot tests take a snapshot of an object the first time they run. This snapshot is committed to version control, and every time the test is run after that, the object is compared to the snapshot. If the snapshot matches the object, the assertion passes. If the snapshot does not match, the assertion fails.
Snapshot testing, unlike standard unit testing, is not a mechanism to detect regressions. The CloudFormation template produced during the synthesis of an AWS CDK app is influenced by both the CDK application code as well as the CDK framework. In some cases, the synthesized template can change when the version of the CDK framework is upgraded. This typically happens when a new best practice gets incorporated into the CDK. For this reason, snapshot testing is best used as a mechanism to alert you when anything at all changes in your CDK stacks. Snapshot testing will make these changes visible to you early.
Refactoring your CDK code is another good use of snapshot testing — you don’t want anything to change while you’re refactoring, and snapshot tests will clearly show you when that happens. For almost all other use cases, fine-grained assertions are a better tool.
You can employ snapshot testing with the assertions module by first synthesizing the stack into a CloudFormation template, converting the entire template to an object (in Java, a Map, in Python, a dict), and then using our test framework’s snapshot testing functionalities to assert that the template matches that snapshot.
from aws_cdk import aws_sns as sns
from aws_cdk import core as cdk
from aws_cdk.assertions import Template
from app.processor_stack import ProcessorStack
# The snapshot parameter is injected by Pytest -- it's a fixture provided by
# syrupy, the snapshot testing library we're using:
# https://docs.pytest.org/en/stable/fixture.html
def test_matches_snapshot(snapshot):
# Set up the app and resources in the other stack.
app = cdk.App()
topics_stack = cdk.Stack(app, "TopicsStack")
topics = [sns.Topic(topics_stack, "Topic1")]
# Create the ProcessorStack.
processor_stack = ProcessorStack(
app, "ProcessorStack", topics=topics # Cross-stack reference
)
# Prepare the stack for assertions.
template = Template.from_stack(processor_stack)
assert template.to_json() == snapshotTesting Constructs
With the assertions module, constructs can be tested similarly to stacks. Since there’s no stack to create, the difference is that we create a stack that holds the construct to be tested.
For example, if we have this DeadLetterQueue construct (previously used in the Testing infrastructure with the AWS Cloud Development Kit blog post):
from aws_cdk import aws_cloudwatch as cloudwatch
from aws_cdk import aws_sqs as sqs
from aws_cdk import core as cdk
class DeadLetterQueue(sqs.Queue):
def __init__(self, scope: cdk.Construct, id: str):
super().__init__(scope, id)
self.messages_in_queue_alarm = cloudwatch.Alarm(
self,
"Alarm",
alarm_description="There are messages in the Dead Letter Queue.",
evaluation_periods=1,
threshold=1,
metric=self.metric_approximate_number_of_messages_visible(),
)You can test it like this:
from aws_cdk import core as cdk
from aws_cdk.assertions import Match, Template
from app.dead_letter_queue import DeadLetterQueue
def test_creates_alarm():
stack = cdk.Stack()
DeadLetterQueue(stack, "DeadLetterQueue")
template = Template.from_stack(stack)
template.has_resource_properties(
"AWS::CloudWatch::Alarm",
{
"Namespace": "AWS/SQS",
"MetricName": "ApproximateNumberOfMessagesVisible",
"Dimensions": [
{
"Name": "QueueName",
"Value": Match.any_value(),
},
],
},
)Summary
Testing is important to ensure that your code does what you expect and prevent regressions and unexpected changes when making changes. The new AWS CDK assertions module provides new and more powerful ways to test your infrastructure as code, especially if you are developing in a language where the assert module isn’t available. For more information on the assertions module, refer to the API reference. As always, we welcome bug reports, feature requests, and pull requests on the aws-cdk GitHub repository.