AWS DevOps Blog

Using AWS Lambda and Amazon DynamoDB in an Automated Approach to Managing AWS CloudFormation Template Parameters and Mappings

AWS CloudFormation gives you an easy way to codify the creation and management of related AWS resources. The optional Mappings and Parameters sections of CloudFormation templates help you organize and parameterize your templates so you can quickly customize your stack. As organizations adopt Infrastructure as Code best practices, the number of mappings and parameters can quickly grow. In this blog post, we’ll discuss how AWS Lambda and Amazon DynamoDB can be used to simplify updates, reuse, quick lookups, and reporting for these mappings and parameters.

Solution overview

There are three parts to the solution described in this post. You’ll find sample code for this solution in this AWS Labs GitHub repository.

  1. DynamoDB table: Used as a central location to store and update all key-value pairs used in the ‘Mappings’ and ‘Parameters’ sections of the CloudFormation template. This could be a centralized table for the whole organization, with a partition key consisting of the team name and environment (for example, development, test, production) and a sort key for the application name. For more information about the types of keys supported by DynamoDB, see Core Components in the Amazon DynamoDB Developer Guide.

Here is the sample data in the table. “teamname-environment” is the partition key and “appname” is the sort key.

{
        "teamname-environment": "team1-dev",
        "appname": "app1",
        "mappings": {
            "elbsubnet1": "subnet-123456",
            "elbsubnet2": "subnet-234567",
            "appsubnet1": "subnet-345678",
            "appsubnet2": "subnet-456789",
            "vpc": "vpc-123456",
            "appname":"app1",
            "costcenter":"123456",
            "teamname":"team1",
            "environment":"dev",
            "certificate":"arn-123456qwertyasdfgh",
            "compliancetype":"pci",
            "amiid":"ami-123456",
            "region":"us-west-1",
            "publichostedzoneid":"Z234asdf1234asdf",
            "privatehostedzoneid":"Z345SDFGCVHD123",
            "hostedzonename":"demo.internal"
        }
}


 

  1. Lambda function: Accepts the inputs of the primary keys, looks up the DynamoDB table, and returns all the key-value data.
  2. Custom lookup resource: Makes a call to the Lambda function, with the inputs of primary keys (accepted by the Lambda function) and retrieves the key-value data. All CloudFormation templates can duplicate this generic custom resource.

 

The diagram shows the interaction of services and resources in this solution.

  1. Users create a DynamoDB table and, using the DynamoDB console, AWS SDK, or AWS CLI, insert the mappings, parameters, and key-value data.
  2. CloudFormation templates have a custom resource, which calls a Lambda function. The combination of team name and application environment (“teamname-environment”) is the partition key input. Application Name (“appname”) is the sort key input to the Lambda function.
  3. The Lambda function queries the DynamoDB table based on the inputs.
  4. DynamoDB responds to the Lambda function with the results.
  5. The Lambda function responds to the custom resource in the CloudFormation stack, with the key-value data.
  6. The key-value data retrieved by the custom resource is then used by other resources in the stack using GetAtt Intrinsic function.

 

Using the sample code

To use the sample code for this solution, follow these steps:

  1. Clone the repository.
git clone https://github.com/awslabs/custom-lookup-lambda.git
cd custom-lookup-lambda
  1. The values in the sample-mappings.json file will be inserted into DynamoDB. Each record in sample-mappings.json corresponds to an item in DynamoDB. The mappings object contains the mapping.
  2. This solution uses Python as a programming model for AWS Lambda. If you haven’t set up your local development environment for Python, follow these steps, and then install the awscli python package.
pip install awscli
  1. To prepare your access keys or assume-role to make calls to AWS, configure the AWS Command Line Interface as described here. The IAM user or the assumed role used to make API calls must have, at minimum, this access. You can attach this policy to the IAM user or IAM group if you are using access keys, or to the IAM role if you are assuming a role. For more information, see Attaching Managed Policies in the IAM User Guide.
  2. Run insertrecord.sh to create the DynamoDB table named custom-lookup and insert the items in sample-mappings.json.
./insertrecord.sh

This script does the following:

  • Installs boto3 and requests Python packages through pip.
  • Executes the Python script to create the DynamoDB table (custom-lookup) and puts the data in sample-mappings.json.
  1. Run deployer.sh to package and create the Lambda function.
./deployer.sh

This script does the following:

  1. Using sample-stack.yaml, create a stack that makes a call to the Lambda function created in step 6. The function queries the DynamoDB table and produces as output the values corresponding to team1-dev and app1.
aws cloudformation deploy --template-file sample-stack.yaml --stack-name sample-stack
  1. Examine the output in the AWS CloudFormation console. You should see the values retrieved from the DynamoDB table. The Fn::GetAtt function allows you to use the values retrieved by the custom resource Lambda function.

For example, if the custom resource Lambda function is called using resource name as CUSTOMLOOKUP in the sample-stack, the value of key=amiid will be used in the stack using !GetAtt CUSTOMLOOKUP.amiid. Likewise, the value of key=vpc will be used in the stack using !GetAtt CUSTOMLOOKUP.vpc and so on.

Conclusion

In this blog post, we showed how to use an AWS CloudFormation custom resource backed by an AWS Lambda function to query Amazon DynamoDB to retrieve key-value data, thereby replacing the Mappings and Parameter sections of the CloudFormation template. This solution provides a more automated approach to managing template parameters and mappings. You can use the DynamoDB table to simplify updates, reuse, quick lookups, and reporting for these mappings and parameters.