AWS Cloud Operations & Migrations Blog

Git pre-commit validation of AWS CloudFormation templates with cfn-lint

We’re living in a golden age of AWS CloudFormation tooling. Tools like cfn_nag and taskcat make it easier to treat your infrastructure as code by performing testing and validation before you deploy a single resource into your accounts. In this blog post, I’ll show you how to use linters to validate your CloudFormation templates.

A linter is a program that examines your code looking for syntax problems or even bugs that can lead to errors when you execute the code. It can be run as a stand-alone tool, but it is more often integrated into your build automation or even your authoring tools.

Historically, CloudFormation linting was limited to the ValidateTemplate action in the service API. This action tells you if your template is well-formed JSON or YAML, but doesn’t do much to validate the actual resources you’ve defined. So, several months ago I went searching for a better option. Tools like Stelligent’s cfn_nag perform additional validation of your template resources, but from a security and best practice perspective. Doing further research, I discovered the CloudFormation service team has been publishing a Resource Specification describing valid syntax for resources and properties.

Introducing the AWS CloudFormation linter

Around the same time, my fellow Amazonian Kevin Dejong was preparing to open source a CloudFormation linter he’d written in Python that checked most of the boxes I was looking for:

  • Intrinsic function parsing (including conditions)
  • An extensible, rule-based architecture
  • Validation against the published resource specification
  • Written in Python (hey, I’m biased!)

I was particularly intrigued by the extensibility of this linter, which allows community members to add new syntax rules for things that exist outside the resource spec, such as allowed values and either/or resource properties.

We published the linter last spring as cfn-lint and have been working to improve it ever since. We’ve added batch file validation, SAM template support, limit checking, and deeper validation for resources like pipelines created using AWS CodePipeline and IAM policies. You can find the full list of linter rules in the GitHub documentation.

The best way to experience the linter is to try it for yourself.

Install the linter

Since it’s written in Python, the easiest way to install the linter is with pip, (assuming you already have Python installed on your machine):

pip3 install cfn-lint

After you have the linter installed and in your operating system path, you can run it from the command line. The basic syntax looks like this:

cfn-lint --template simple-vpc.template.yml --region us-east-1 --ignore-checks W

This tells the linter to validate the simple VPC template against the resource specification in the us-east-1 Region, ignoring all warning rules.

But the linter supports a number of use cases beyond just the command line:

  1. As a pre-commit hook for your Git repository.
  2. As an IDE plugin for Atom, Visual Studio Code, Sublime, IntelliJ, and even VIM.
  3. As a build/validation step in your CI/CD pipeline.

In this blog post, I’m going to walk you through setting up cfn-lint as a pre-commit hook for Git. I’ll focus on the other use cases in future articles.

Set up your Git repository

First you’ll need a Git repository for your templates. If you don’t have one already, you can clone my demo repo:

git clone https://github.com/cmmeyer/cfnlintdemo.git

You can also create an empty repository:

mkdir cfnlintdemo
cd cfnlintdemo
git init

Configure cfn-lint as a pre-commit hook

Next, you’ll need to install pre-commit and associate it with Git. If you’re not familiar with the pre-commit framework you can read more on their website. It provides a way to execute scripts to identify simple issues in your code before submission for code review. In this case, to validate the syntax of your CloudFormation templates.

You can also install pre-commit via Python’s pip installer:
sudo pip install pre-commit

Alternatively, you can use the Python-based installer that they supply on their website:
curl https://pre-commit.com/install-local.py | python -

Next, you’ll need to tell pre-commmit to use cfn-lint as a hook. You do this by creating a .pre-commit-config.yaml file in the root of your repository. Here’s the one I have in my cfnlintdemo repo:

# .pre-commit-config.yaml
repos:
-   repo: https://github.com/awslabs/cfn-python-lint
    rev: v0.15.0 # The version of cfn-lint to use
    hooks:
    -   id: cfn-python-lint
        files: templates/.*\.(json|yml|yaml)$

This tells pre-commit to install cfn-lint version 0.15.0 (the latest at the time I was writing this), and run it against all the files in the templates directory whenever there is a git commit action.

The last step is to install the hook in your repository by running the following command from the root of your repo:

pre-commit install

Try to add a bad template

Now that the linter is installed, you can test it by attempting to commit a bad template to your repo. This example template has a bad subnet route table association that could cause real problems when you attempt to update or delete the stack:

# cfnlintdemo/templates/bad-routeable-association.yaml
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Bad SubnetRouteTableAssociation'

Parameters:
  PublicRouteTable:
    Type: String

  PrivateRouteTable:
    Type: String

  PublicSubnet01:
    Type: String

  PrivateSubnet01:
    Type: String


Resources:
  PublicSubnetRouteTableAssociation1:
      Type: AWS::EC2::SubnetRouteTableAssociation
      Properties:
        RouteTableId: {Ref: PublicRouteTable}
        SubnetId: {Ref: PublicSubnet01}

  PrivateSubnetRouteTableAssociation1:
      Type: AWS::EC2::SubnetRouteTableAssociation
      Properties:
        RouteTableId: {Ref: PrivateRouteTable}
        SubnetId: {Ref: PublicSubnet01}

Save the file, then go ahead and try to commit it to your repo.

git add /templates/bad-reoutetable-association.yaml
git commit -m "Bring the badness"

You should see results similar to the following:

[INFO] Installing environment for https://github.com/awslabs/cfn-python-lint.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
AWSLabs CloudFormation Linter............................................Failed
hookid: cfn-python-lint

W2001 Parameter PrivateSubnet01 not used.
templates/bad-route-table-association.yaml:15:3

E3022 SubnetId in PublicSubnetRouteTableAssociation1 is also associated with PrivateSubnetRouteTableAssociation1
templates/bad-route-table-association.yaml:24:9

E3022 SubnetId in PrivateSubnetRouteTableAssociation1 is also associated with PublicSubnetRouteTableAssociation1
templates/bad-route-table-association.yaml:30:9<

Conclusion

Congratulations! You’ve successfully prevented this bad template from being committed to your Git repository. You can now add cfn-lint as a pre-commit hook for any Git repository that contains CloudFormation templates.

About the Author

Chuck Meyer is a Senior Developer Advocate for AWS CloudFormation based in Ohio.  He spends his time working with both external and internal development teams to constantly improve the developer experience for CloudFormation users.  He’s a live music true believer and spends as much time as possible playing bass and watching bands. Contact him on Twitter (@chuckm) if you’d like to join our #cloudformation Slack channel.