Deep Dive on AWS CloudFormation Macros to transform your templates

AWS CloudFormation macros add custom operations to your templates, including iterations, string manipulations, and math operations. Macros allow these language extensions without sacrificing the declarative benefits enjoyed by our customers, whether they are novice developers or experienced system admins.

CloudFormation macros are ideal for system administrators and developers who benefit from the additional logic to write cleaner CloudFormation templates, avoid code duplication, and centrally maintain repeatable actions across many templates. Further, as new developers learn CloudFormation, macros can provide an initial customization option.

This post is the first in a two-post series that covers use cases for CloudFormation macros, why macros evolved as a necessary addition to the service, and how to use macros for maximum benefit. In this post, I explain how macros enable CloudFormation to do something beyond its native functionality. In the second post, I show you how to write a macro and deploy it.

Why macros?

AWS CloudFormation is declarative by nature. You describe the things you want it to create in a JSON- or YAML-formatted template and it creates the desired cloud resources for you. Most declarative coding approaches have an imperative element to them and the CloudFormation configuration language is no exception. For example, when you launch an Amazon Elastic Compute Cloud (Amazon EC2) instance, you have the option to pass user data to the instance that can be used to run configuration tasks and scripts after the instance starts. CloudFormation allows conditional statements in the form of if-then clauses and other simple intrinsic functions. Although helpful and at times even critical, they don’t replace the flexibility of modern languages like Java or Python. The CloudFormation configuration language is purposefully kept simple so that it’s easy to read, even by people who aren’t software engineers. It is critical to allow the automation of dependent yet heterogeneous resources. Because CloudFormation is widely used, many customers choose to make it more imperative using tools like the AWS Cloud Development Kit (AWS CDK), Troposphere, Jinja, and others to expand on its built-in functions. These tools are typically used for two purposes:

• To inject imperative instructions while primarily working in CloudFormation’s declarative code.
• To write the entire template in a common, high-level, imperative language, and then translate that language into CloudFormation’s native code.

When you become a CloudFormation expert, you might want to explore these alternatives. For the first category, where you optimize and balance the use of both declarative and imperative code, your options are:

• JSON/YAML: This is the standard way to declare resources in an easy to understand, lightweight syntax. Most CloudFormation customers use this option when requirements are simple enough to warrant small- and medium-sized templates.
• Basic transforms: CloudFormation provides two transforms by default. They are essentially AWS Lambda functions maintained by the CloudFormation backend. Customers requested these basic, use as-is transforms several years ago. Because of their simplicity, they are useful to lots of customers, no matter where they are on their cloud journey. The two transforms are,

AWS::Serverless transform

Popular and widely used, this transform takes an entire template written in the AWS Serverless Application Model (AWS SAM) syntax and expands it into a compliant AWS  CloudFormation template.

Although the transform code is not open source, the model code and utilities are open source, so customers and contributors can continue to influence its functionality.

Because AWS SAM is a mature project, many customers and contributors have driven the creation of several utilities. For example, sam-local simplifies the writing, deployment, and maintenance of serverless workloads by prescribing a development workflow that can use common editors and IDEs running on developer environments.

AWS::Include transform

This transform takes a snippet of template code from a file stored in an Amazon Simple Storage Service (Amazon S3) bucket and pastes it into any template. The transformation is completed before template parsing and execution (create stack or update stack).

You can declare common resources used across many stacks in one template snippet and reference the snippet in many templates. You can maintain this common code in one place and reuse the configuration.

This transform is designed to be simple. Although it might sound like it is equivalent to keywords like include or import in imperative languages like Python, it’s more like simple text substitution (copy and paste).

          You can reference other tranforms here.

• Advanced transforms: When macros were introduced in September 2018, AWS CloudFormation extended the idea of the basic transform. Macros now enable customers to create their own transforms. They can decide how simple or complex their custom transformation logic will be, based on their requirements. Macros offer customers advanced ways to control how their template code will be transformed.

AWS also provides a sophisticated option for writing templates in high-level languages and then translating them into CloudFormation templates. The AWS CDK is a popular, multi-language, open-source framework that enables developers to harness the full power of modern programming languages to define cloud infrastructure and provision applications using AWS CloudFormation. Because of its popularity, the AWS CDK can now do more than just translate to CloudFormation code. For information about the latest features, see the AWS CDK documentation on GitHub.

Learning macros by example

There are two major steps to creating and using a macro. First, create a macro resource definition. This definition includes the reference to an AWS Lambda function that performs the required template transformation. Define it as a resource of type AWS::CloudFormation::Macro so users can reference the Lambda function from the templates.

MacroDefinition:    
  Type: AWS::CloudFormation::Macro
  Properties:
    Name: SubscriptionFilterMacro
    Description: Macro to create subscription filters for CloudWatch Logs
    FunctionName: !GetAtt TransformFunction.Arn

Second, reference the macro in your template. You have two options here:

• If you want to process just a subsection of a template, use Fn::Transform.
• If you want to process the entire template, reference the macro in the Transform section of the template. This section is optional. It is declared at the same level as other primary template sections like Parameters and Resources.

This post includes examples to help you understand the different macro use cases. The code on the left (column1) represents the invocation of the macro before processing. The code on the right (column3) shows the processed template, resulting from the macro invocation, that is then submitted to the AWS CloudFormation for CREATE or UPDATE operations. You’ll find the macro code for the examples on CloudFormation macros GitHub.

Example 1: Loop or iterate to repeat resources

This example shows how we can use a macro to iteratively create any number of resources. The macro looks for the Count keyword, retrieves its parameter’s value (in example, 3), and creates three resources. This example also illustrates how you can pass runtime parameters to the macro.

Example 2: Manipulating strings

This example shows how you can incorporate string operations using macros. It converts the text to uppercase or lowercase, determine the length of a string, retrieve a substring while removing special characters, among other options. [PH1] The macro invocation results in the conversion of the InputString value to uppercase. It shows how the parameters are passed. The logic capability is extended by passing both a string parameter and the desired operation on the string.

Example 3: Global variables

This example shows how you can mimic a global variable concept in your templates and recall them with a custom syntactical convention. GlobalsMacro replaces all text mentions preceded by the @ symbol, enabling a more reusable template. All future changes to the variables can be performed in the Globals section of templates using the macro. The ‘@’ symbol is not a reserved keyword in CloudFormation.

Other CloudFormation extensions

You can extend the functionality of CloudFormation through private resource types and custom resources, which allow you to call custom code written in imperative languages such as Python, Java, and so on. With a private resource type, you can create your own CloudFormation resource type and register it in the AWS CloudFormation registry. During re:Invent 2020, CloudFormation released modules, which help you package resource configurations that can be reused across CloudFormation stacks.

Conclusion

In this blog post, I showed you how macros can be used to go beyond CloudFormation’s built-in declarative language and make your CloudFormation templates more versatile. In the second post of this two-part series, I show you how to create a macro.

For information about other aspects of macros, including event mappings, evaluation order, and more, see creating an AWS CloudFormation macro definition in the AWS CloudFormation User Guide. You can find macro examples on GitHub. Please contribute and share your macros with the CloudFormation community.

About the Author

Wilson Dayakar Puvvula is a Cloud Applications Architect at AWS. He helps customers adopt AWS services and offers guidance for AWS automation and cloud-native application implementations. Learning new things and building solutions at AWS scale is what excites him. Outside of work, he watches Liverpool F.C. and enjoys the team anthem, You’ll Never Walk Alone. You can find him on twitter at @PuvvulaWilson