Integration & Automation

8 best practices when automating your deployments with AWS CloudFormation

If you’re automating your workload deployments on the Amazon Web Services (AWS) Cloud using AWS CloudFormation, you can take steps to save time during and after initial development. In addition to saving time, you can prevent your templates from becoming obsolete. In this blog post, I cover some best practices for AWS customers and AWS Partners to follow when developing infrastructure as code with AWS CloudFormation. The post also covers best practices for testing and maintaining AWS CloudFormation templates.

About this blog post
Time to read 9 min.
Learning level Intermediate (200)
AWS service AWS CloudFormation

About AWS CloudFormation

AWS CloudFormation streamlines the deployment of key workloads on the AWS Cloud. With AWS CloudFormation, you model and provision all the resources needed for your applications across multiple Regions and accounts in an automated and secure manner. You can use programming languages or simple text files. AWS CloudFormation templates are text files, written in YAML or JSON format, that define the AWS resources to be deployed.

You can integrate automation tools, such as AWS Lambda and AWS Systems Manager, into your AWS CloudFormation templates. With automation tools, you can automate the provisioning of custom workloads on top of the defined AWS infrastructure.

The AWS Quick Start program provides over 170 examples of AWS Partner and AWS native deployments, all of which use AWS CloudFormation templates.

Best practices

  1. Start with existing AWS CloudFormation templates
  2. Create modular templates
  3. Use existing repositories as submodules
  4. Use an integrated development environment with linting
  5. Use parameters to define paths to your external assets
  6. Use the same names for common parameters
  7. Automate AWS CloudFormation testing with TaskCat
  8. Maintain your templates

1. Start with existing AWS CloudFormation templates

When you use AWS CloudFormation to develop infrastructure as code, first check to see if what you plan to deploy has already been developed and shared with the community. Check for your entire solution as well as aspects of it.

For example, look at AWS Quick Starts, which are open-source and available on GitHub. On the AWS Quick Starts home page, you can search the catalog by partner, product, or keyword, and you can filter by common use cases. Also look at AWS Labs, another open-source GitHub organization with a number of sample AWS CloudFormation templates developed by AWS and by the community of AWS Partners and customers.

If you’re building an automated deployment on AWS infrastructure that includes common AWS resources, starting with an existing template can save you hours or days worth of effort. Common resources include Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Simple Storage Service (Amazon S3) buckets, Lambda functions, and Amazon Relational Database Service (Amazon RDS) databases.

2. Create modular templates

While you could use a single AWS CloudFormation template to automate a complex deployment, it’s easier to work with multiple smaller templates. For example, let’s take a three-tier web application with the following parts:

  • A highly available Amazon RDS database
  • An Amazon EC2 Auto Scaling group for the application layer
  • An Amazon EC2 Auto Scaling group for the web layer
  • Load balancers for both the application and web layers

If you were to define every part of this deployment in a single AWS CloudFormation template, the template would be large and difficult to troubleshoot. Troubleshooting and iterating would be easier if, instead, you created a template for each of these components. You would have a child template for the database, a child template for the application layer, a child template for the web layer, and a parent template that deploys the child templates in sequence.

Using a modular approach like this, you and others could reuse the smaller templates. For example, you might copy the database template and use it for another web application.

3. Use existing repositories as submodules

main repository with two submodules and a sub-submoduleUse submodules to take advantage of work that has already been done. Submodules not only save you time during development but also reduce the need to maintain redundant resources.

If you’re maintaining your AWS CloudFormation templates in a GitHub repository, you can import other GitHub repositories as submodules and deploy them from your parent AWS CloudFormation template. To do this, use the git submodule add command.

The Amazon Virtual Private Cloud (Amazon VPC) Quick Start is a good example. If you’re deploying infrastructure that requires a new VPC, you can add the Amazon VPC Quick Start to your repository as a submodule. Either fork the quickstart-aws-vpc repo to your GitHub account or clone it directly from the aws-quickstart account. Then, your parent stack can call the submodule template as a child stack, passing parameters into it to define the VPC. Parameters include the number of Availability Zones to use, whether to deploy network address translation (NAT) gateways, etc.

Other common submodules include the Linux bastion Auto Scaling group, Microsoft Active Directory Domain Services, and Amazon Elastic Kubernetes Service (Amazon EKS).

4. Use an integrated development environment with linting

The amount of time it takes to launch an AWS CloudFormation template varies from a few minutes to several hours. With complex deployments, a developer might make a simple mistake and not realize it until a stack fails late during testing. Discovering mistakes at he last minute can lead to unnecessary frustration and wasted time.

While AWS CloudFormation templates and scripts can be developed with any text editor, working in an integrated development environment (IDE) can improve the process. An IDE can catch formatting mistakes in real time, can display aspects of your code in different colors to highlight them, and can reformat multiple lines of code simultaneously. Most IDEs can incorporate third-party linting tools to ensure that your code is developed correctly for particular applications, including AWS CloudFormation.

AWS CloudFormation Linter (cfn-lint) is an open-source tool maintained by the AWS CloudFormation team. Cfn-lint analyzes AWS CloudFormation templates and checks for syntactic errors. (The cfn in cfn-lint stands for CloudFormation.) By using an IDE with this linter installed, developers can catch errors immediately, reducing development and testing time. You can find cfn-lint plugins for popular IDEs on the CloudFormation Linter GitHub page under Editor Plugins.

5. Use parameters to define paths to your external assets

Future-proof your AWS CloudFormation templates by avoiding hardcoded paths to external assets. For example, imagine that you have developed a template that deploys an EC2 instance that downloads installation media from an S3 bucket. You could hardcode the path to the installation media as something like this:

https://customerbucket.s3.us-east-1.amazonaws.com/install.tar.gz

If you did, though, future changes to the S3 bucket (in this case, customerbucket)—such as removing public access or deleting the bucket entirely—would cause the deployment to fail.

Instead of hardcoding the path to the installation media, define it as a parameter, and set the URL as the default.

Another way to future-proof access to your external assets is to store them in the same repository as your AWS CloudFormation templates. You can store external assets that are called by the AWS CloudFormation template in the same repository as the template, decreasing external dependencies. Common repositories include AWS CodeCommit, GitHub, and GitLab. Assets that are commonly stored alongside the AWS CloudFormation templates that call them include Lambda functions, Bash or PowerShell scripts, and installation media.

6. Use the same names for common parameters

When you use a modular approach with complex deployments, you may find yourself working with a large number of AWS CloudFormation templates. The Amazon EKS Quick Start, for example, has 15 templates. In the Amazon EKS Quick Start, outputs from one template are passed as parameter values to other templates. Keeping parameter names consistent across all AWS CloudFormation templates makes it easier to troubleshoot and iterate.

For example, take the parameter VPCID (the ID of VPC into which the Quick Start is deployed). If you were to deploy the Amazon VPC Quick Start as a submodule, then you would deploy assets into the new VPC by passing the VPCID output from the submodule as a parameter value to your other templates. Using the parameter name VPCID in all your templates simplifies keeping track of your outputs and parameter values.

Here are some other common parameter names to keep the same:

  • VPCCIDR – Classless Inter-Domain Routing (CIDR) block of the VPC into which the Quick Start is deployed
  • PrivateSubnet1ID – ID of the first Availability Zone’s private subnet
  • PrivateSubnet2ID – ID of the second Availability Zone’s private subnet
  • PublicSubnet1ID – ID of the first Availability Zone’s public subnet
  • PublicSubnet2ID – ID of the second Availability Zone’s public subnet

7. Automate AWS CloudFormation testing with TaskCat

When you build AWS CloudFormation templates, you must test them. Testing typically involves the following steps:

  1. Upload your templates to an S3 bucket.
  2. Sign in to the AWS Management Console.
  3. Open the AWS CloudFormation console.
  4. Enter the S3 path to your parent template.
  5. Manually enter parameter values.
  6. Launch the stack.
  7. Wait to see if the deployment succeeds or fails.

It’s time-consuming to manually deploy AWS CloudFormation templates across multiple AWS Regions and then clean up the assets you deployed during testing. In addition, you may lose track of the assets you deployed. So when you clean up the test environment manually, you may overlook assets that remain active in your account. These overlooked assets can lead to unexpected charges on your next AWS billing statement.

To address these issues, the AWS Quick Start team has developed a process to automate AWS CloudFormation testing using an open-source tool called TaskCat. You enter Regions and parameter values into a YAML-formatted file called taskcat.yml in the root of your repository. TaskCat uses these values to automatically deploy your AWS CloudFormation templates. It then logs the results of the deployment—whether it succeeded or failed (and, if it failed, the reason)—and deletes any assets that were deployed.

When you use TaskCat to test your AWS CloudFormation templates, you can focus on other things while waiting for the deployment to complete. You can also prevent resources that were deployed during testing from lingering in your AWS accounts.

8. Maintain your templates

Since AWS CloudFormation templates are used to automate deployments, they are often reused for months or years. As a result, they can get out of date. For example, EC2 instance types may become deprecated, Amazon Machine Images (AMIs) may be removed from general availability, and new functionality may be added to a database service.

Test your AWS CloudFormation templates periodically to ensure that they continue to work as expected. Incorporate new service functionality as it becomes available.

Conclusion

As an AWS CloudFormation developer, you can save time, effort, and frustration when you follow the best practices described in this post. For more, see the following resources:

Let me know your thoughts in the comments, or send an email to quickstart@amazon.com.