Four ways to retrieve any AWS service property using AWS CloudFormation (Part 2 of 3)

This post is the second in a series on how to build customizations using AWS CloudFormation. In part 1, we showed you how to develop customizations using cfn-response and crhelper and shared the scenarios they are best suited for. In this post, we’ll use AWS CloudFormation macros to address some of the coverage gaps identified in our public roadmap.

Prerequisites

To complete the example presented here, we use YAML, AWS Lambda, and Python. We also use the AWS Serverless Application Model (AWS SAM), a built-in transform hosted by AWS CloudFormation that provides shorthand syntax to express functions, APIs, and other resources.

Option 3: Custom resource using AWS CloudFormation macros

In part 1, we addressed the request in GitHub issue 157 to retrieve the Amazon EC2 security group name with the cfn-response module. In this post, we’ll use macros to fetch the same security group name. There are about 70 similar issues as of the time of this writing in our public coverage roadmap, collecting more than 168 reactions from the community.

Macros are AWS Lambda-backed functions that you register in AWS CloudFormation to perform code transformations, with operations like string manipulation, find and replace, loops, or other substitutions. Macros are invoked using the Transform keyword in your template. Once the template is uploaded to AWS CloudFormation, and the code parser encounters the Transform keyword, the AWS Lambda function containing your transformation logic is executed . Your macro will be processed after you submit your template for processing, but before starting the CREATE, UPDATE, and DELETE stack operations.

After your macros are deployed, template users aren’t distracted by the macros’ code, so you get the same flexibility in adding custom logic as you did before, while abstracting the details away. Because macros are not bound to a specific template, you can reuse them across many stacks in your account, much like crhelper. In contrast to other templating languages like Jinja, macros are built into the AWS CloudFormation backend. You can use any language Lambda functions support, including those enabled by Lambda layers.

We author AWS CloudFormation templates to define the macro and its Python code. Because we are using AWS SAM, we need to organize our code and templates in a specific way. We create the following directory structure:

$ tree option3
option3
├── cfn-ec2-custom-resource.yml
├── cfn-macro.yml
└── lambda_cr_function
         ├── crhelper
         ├── crhelper-2.0.6.dist-info
         ├── tests
         ├── macro.py
         └── resource.py

Let’s look at how macros are invoked in templates. Pay close attention to the following sample template, because it hints at how we’ll use a new macro to retrieve the attribute we are missing:

cfn-ec2-custom-resource.yml

AWSTemplateFormatVersion: '2010-09-09'

Transform: SGMacro

Parameters:
  vpcID:
    Type: AWS::EC2::VPC::Id
    Description: Enter VPC Id
    
Resources:
  CfnEC2SecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: CFN2 Security Group Description
      VpcId: !Ref vpcID
  
  CustomSecurityGroupNameMacro:
    Type: SGMacro    
    Properties:
      RefId: !Ref 'CfnEC2SecurityGroup'
      
Outputs:
  SecurityGroupID:
    Description: Security Group ID
    Value: !Ref CfnEC2SecurityGroup
  SecurityGroupName:
    Description: Security Group Name from Macro
    Value: !GetAtt 'CustomSecurityGroupNameMacro.SecurityGroup-Name'

What the template does

We use the Transform keyword to signal to AWS CloudFormation that we’re using a macro named SGMacro. This keyword is at the same global level as the Parameters, Resources, and Outputs keywords. We use Parameters to capture the VPC ID, making this template reusable. In the required Resources section, we create an AWS::EC2::SecurityGroup in the VPC.

Next, we invoke SGMacro to get the missing EC2 security group name. There are two ways to reference a macro in your template. In our example, we use the Transform keyword at the top of the file to allow SGMacro to process the entire template content and to invoke it multiple times, if necessary. If you want to scope down the template content that a macro can transform, use the Fn:Transform intrinsic function inside any resource definition. You can use both conventions in the same template for different macros. If you do, see the evaluation order logic in the AWS CloudFormation User Guide.

We have defined our SGMacro to expect an EC2 security group ID as a parameter and to return the EC2 security group name when executed. This is one of the big advantages to using macros: end users can invoke them much like they declare resources, without worrying about your function code.

Using the !Ref and !GetAtt intrinsic functions, we display the EC2 security group ID and name in the Outputs section.

Now that we know what the structure and syntax of our template looks like, let’s build the parts that will enable us to use it in its simplified format.

Create macros with AWS SAM

To create a macro, we must first define and then implement it. We use the open source AWS SAM framework.

cfn-macro.yml

AWSTemplateFormatVersion: '2010-09-09'

Transform: AWS::Serverless-2016-10-31

Resources:
  ResourceFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.8
      CodeUri: lambda_cr_function
      Handler: resource.handler
      Timeout: 30
      Policies:
      - Statement:
        - Sid: CloudWatchLogPolicy
          Effect: Allow
          Action:
          - logs:CreateLogGroup
          - logs:CreateLogStream
          - logs:PutLogEvents
          Resource: '*'
        - Sid: EC2DescribePolicy
          Effect: Allow
          Action:
          - ec2:DescribeSecurityGroups
          Resource: '*'

  MacroFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.8
      CodeUri: lambda_cr_function
      Handler: macro.handler
      Environment:
        Variables:
          LAMBDA_ARN: !GetAtt ResourceFunction.Arn

  Macro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: SGMacro
      FunctionName: !GetAtt MacroFunction.Arn 

What the template does

First, using the shorthand syntax we mentioned earlier, we signal to AWS CloudFormation that our code requires the AWS::Serverless transform to create two required functions. The syntax is similar to the AWS::Lambda::Function type, but it simplifies the definition of the function and uploads all required artifacts to Amazon Simple Storage Service (Amazon S3) before deployment.

In the Resources section, we start by defining ResourceFunction, an AWS Lambda function to get the missing EC2 security group name attribute. This resource is configured with security controls to allow only ec2:DescribeSecurityGroups API actions and to log data to Amazon CloudWatch.

The use of AWS SAM here is deliberate, because it shows the power of using multiple macros to create clear, concise code while enabling customization. Because we used AWS SAM, we don’t need to explicitly define AWS IAM roles and policies separately in our code, not even the S3 bucket we use behind the scenes! This is why AWS SAM is so popular. Learning macros can empower you to add similar coding efficiencies for both developers and end users. Aren’t you glad you are reading this?

Taking advantage of the same AWS SAM automations, we implement the wrapper by defining it as another AWS Lambda function, MacroFunction. This function expects the entire AWS CloudFormation template as a JSON payload request. MacroFunction then parses and transforms the payload.

Our last resource is where we declare the macro using AWS::CloudFormation::Macro, which allows it to be used in other templates. The name of the macro must be a unique resource identifier in the Region and account that implements it. When our SGMacro is invoked, it also expects the entire AWS CloudFormation template passed in as a JSON payload. For an in-depth macro workflow, see the Extending AWS CloudFormation with AWS Lambda Powered Macros blog post.

Functions

This next section is exactly the same as outlined in part 1 of this series, but to make each of these options stand alone, we repeat it here for your convenience. We first install the crhelper Python module like we did before, and then we create the AWS Lambda function.

mkdir lambda_cr_function
cd lambda_cr_function
#install crhelper in current directory using --target flag
pip install --target . crhelper

resource.py

from __future__ import print_function
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger()
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL')
@helper.create
@helper.update
def create(event, context):
    # 1. retrieve resource reference ID or Name
    ResourceRef = event['ResourceProperties']['ResourceRef']    
    # 2. retrieve boto3 client    
    client = boto3.client('ec2')
    # 3. Invoke describe/retrieve function using ResourceRef
    response = client.describe_security_groups(GroupIds=[ResourceRef['RefId']])
    # 4. Parse and return required attributes 
    helper.Data['SecurityGroup-Name'] = response.get('SecurityGroups')[0].get('GroupName')
    
@helper.delete
def no_op(_, __):
    pass
def handler(event, context):
    helper(event, context)

As you might recall from part 1, crhelper handles timeouts, provides detailed logging, and exposes prebuilt decorators for CREATE, UPDATE, and DELETE requests. The only difference between this code fragment and the one used in part 1 is the use of the describe_security_groups EC2 API action, rather than the describe_file_systems Amazon FSx for Windows File Server API action.

Next, we link the ResourceFunction and MacroFunction together. Our goal is to invoke the ResourceFunction, which has the customization we just built.

Here’s our final piece of code:

macro.py

import os

PREFIX = "SGMacro"

LAMBDA_ARN = os.environ["LAMBDA_ARN"]

def handle_template(request_id, template):
    for name, resource in template.get("Resources", {}).items():
        if resource["Type"].startswith(PREFIX):
            customResponse = resource.update({
                "Type": "AWS::CloudFormation::CustomResource",
                "Properties": {
                    "ServiceToken": LAMBDA_ARN,
                    "ResourceRef": resource.get("Properties", {}),
                },
            })

    return template

def handler(event, context):
    fragment = event["fragment"]
    status = "success"
    try:
        fragment = handle_template(event["requestId"], event["fragment"])
    except Exception as e:
        status = "failure"
    return {
        "requestId": event["requestId"],
        "status": status,
        "fragment": fragment,
    }

During stack creation, MacroFunction returns a JSON-formatted response with requestId, status, and fragment. The fragment contains the transformed template. The following table shows the original and processed template, where SGMacro is replaced by the custom resource function definition, hiding the underlying implementation details from end users.

You can view both the original and processed templates in the console. Using the Template tab on the Stack Details page, you can toggle between code listings, as shown in Figure 1.

Figure 1: Using the Template tab on the Stack Details page

Now that you have seen all the code, it’s time to deploy our macro!

1.  Save the template files under the option3 folder and the Python files under the lambda_cr_function folder.
2. Create a unique S3 bucket to store the AWS CloudFormation artifacts.
3. Use the AWS CLI to run the package command, which packages and uploads local artifacts to the S3 bucket you just created. This command returns a template named cfn-packaged-macro.yml and replaces all references to local artifacts with the S3 bucket location.
4. Use the AWS CLI to run the deploy command.

# Run below commands from the option3 folder
aws s3 mb s3://unique-s3-bucket
aws cloudformation package --template-file cfn-macro.yml --s3-bucket unique-s3-bucket --output-template-file cfn-packaged-macro.yml
aws cloudformation deploy --template-file cfn-packaged-macro.yml --capabilities CAPABILITY_IAM --stack-name simple-sg-macro

Figure 2 shows the successful creation of resources related to SGMacro. This macro is now available to use in any stack in the account and region it was deployed at.

Figure 2: Resources tab on the Stack Details page of the AWS CloudFormation console

Test the macro

Let’s put our macro to the test by using the cfn-ec2-custom-resource.yml template to deploy an AWS CloudFormation stack. The stack creates the EC2 security group and uses SGMacro to retrieve its name.

#deploy the stack
aws cloudformation deploy --stack-name securitygroup-retriever-example --template-file cfn-ec2-custom-resource.yml --parameter-overrides vpcID=vpc_id_in_your_account

#validate stack output 
aws cloudformation describe-stacks --stack-name securitygroup-retriever-example --query "Stacks[0].Outputs"
[
    {
        "OutputKey": "SecurityGroupName",
        "OutputValue": "securitygroup-retriever-example-CfnEC2SecurityGroup-ABCDE1234512",
        "Description": "Security Group Name from Macro"
    },
    {
        "OutputKey": "SecurityGroupID",
        "OutputValue": "sg-1234EXAMPLE4567",
        "Description": "Security Group ID"
    }
]

That’s it! We used macros to retrieve the missing EC2 security group name! We hope this example demonstrates the benefits of using macros. They are reusable, easy to invoke and reference. The built-in AWS SAM transform makes authoring macros even easier. Although you can implement equivalent functionality with crhelper (as we did in part 1), using macros does not require you to account for properly responding to all operation events in your Python code. Moreover, crhelper does not allow you to modify the template before you submit it for CREATE, UPDATE, and DELETE stack operations.

Note: If you use the console or the create-stack and update-stack AWS CLI commands, your template cannot be larger than 51,200 bytes. You can avoid this restriction by using AWS SAM, as we’ve done here.

Cleanup

Use the following commands to delete the resources we created in this option.

aws cloudformation delete-stack --stack-name securitygroup-retriever-example
aws cloudformation delete-stack --stack-name simple-sg-macro
aws s3 rm s3://unique-s3-bucket

Conclusion

In this post, we shared an example of how to use AWS CloudFormation macros to customize AWS CloudFormation. Macros allow you to add utilities and transformations before templates are processed, and those utilities can go beyond the scenarios used in the examples in this series. For more information, see Using AWS CloudFormation macros to perform custom processing on templates in the AWS CloudFormation User Guide and the examples on GitHub.

In part 3 of this series, we’ll conclude with a deep dive into AWS CloudFormation resource types and how you can use them to retrieve any AWS service property using AWS CloudFormation. Check out the other posts in this three-part series:

• Part 1: cfn-response, crhelper
• Part 3: Resource Types

We’ve included our social media contact info below, so please reach out with any comments or questions. We look forward to hearing from you!

About the Author

Gokul Sarangaraju is a Senior Technical Account Manager at AWS. He helps customers adopt AWS services and provides guidance in AWS cost and usage optimization. His areas of expertise include delivering solutions using AWS CloudFormation, various other automation techniques. Outside of work, he enjoys playing volleyball and poker – Set, Spike, All-In! You can find him on twitter at @saranggx.

Luis Colon is a Senior Developer Advocate at AWS specializing in CloudFormation. Over the years he’s been a conference speaker, an agile methodology practitioner, open source advocate, and engineering manager. When he’s not chatting about all things related to infrastructure as code, DevOps, Scrum, and data analytics, he’s golfing or mixing progressive trance and deep house music. You can find him on twitter at @luiscolon1.