AWS Developer Blog

Migrating CloudFormation templates to the AWS Cloud Development Kit

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion. You can use AWS CloudFormation’s sample templates or create your own templates to describe the AWS resources, and any associated dependencies or runtime parameters, required to run your application. You don’t need to figure out the order for provisioning AWS services or the subtleties of making those dependencies work. CloudFormation takes care of this for you. After the AWS resources are deployed, you can modify and update them in a controlled and predictable way, in effect applying version control to your AWS infrastructure the same way you do with your software.

In July 2019, the AWS Cloud Development Kit (CDK) was launched providing even more customer choice by enabling developers to define AWS Infrastructure as Code in familiar programming languages like TypeScript, JavaScript, Python, C#, and Java. The CDK enables you to define your infrastructure using higher-level abstractions, but still leverages CloudFormation under the hood; this means you still get all of CloudFormation’s benefits, including automatic rollback and drift detection.

Many developers want to use the CDK, because they prefer defining infrastructure in the same familiar programming language that they use for their application code. But they already have an extensive library of CloudFormation templates that encode proven infrastructure patterns and critical configuration settings. These templates take time and effort to build and maintain, and manually writing the CDK code that will reproduce these existing stacks in a CDK app is tedious and error-prone.

Until today, the only way to leverage the infrastructure defined in existing templates was the CfnInclude class from the core module, which allowed including a CloudFormation template in a CDK application, and outputting it unchanged. However, this solution has a couple limitations:

  • It does not allow modifying the resources that are part of the template in any way after it has been included.
  • Referencing resources from the template is done by manually using their logical IDs. This can be error prone, and references created in this manner can not participate in CDK’s automatic cross-stack reference generation, which is a powerful capability the CDK offers.

Announcing the CloudFormation-Include CDK module

We are pleased to announce the Developer Preview release of the new cloudformation-include CDK module which was specifically developed to help migrate existing CloudFormation stacks to CDK code. This module parses a CloudFormation template file and loads all elements it finds in the template (resources, parameters, outputs, etc.) into your CDK application. You can then modify any objects defined in the template directly in your CDK code, and reference the existing template resources when creating new CDK constructs.

Prerequisites

  • An AWS account
  • A local CDK installation

Migration

First, you need to have the CDK CLI installed locally. If you don’t have it already, see the instructions in the AWS CDK Developer Guide. To verify your installation works correctly, run the command cdk --version in a terminal; you should see output similar to:

$ cdk --version
1.63.0 (build 7a68125)

You will be using CDK in TypeScript in this blog post; however, the cloudformation-include module is available in all CDK-supported languages, and the code in the other languages follows a similar structure to the TypeScript code. For reference, here’s the documentation of the cloudformation-include module for the other languages:

Next, you’ll need a CloudFormation Stack to migrate to the CDK. For the purposes of this blog post, we’ve created a very simple “migration stack” CloudFormation template which contains a single S3 bucket.

{
  "Resources": {
    "MyBucket": {
      "Type": "AWS::S3::Bucket"
    }
  }
}

If you would like to follow along with the stack migration example in this blog post, you can create a stack with the JSON block above, or click the following link while you’re logged into your AWS account to deploy the migration stack automatically. This will open the CloudFormation new Stack creation wizard, feel free to accept all of the default options and step through the wizard until the CloudFormation stack is created.

Deploy the migration stack to my AWS account

Now that you have a CloudFormation stack in your account, you need to create a new CDK application in a directory called migration. To do this, open a terminal and run the following commands:

$ mkdir migration
$ cd migration
$ cdk init -l typescript

The next thing you need to do is add a dependency on the cloudformation-include module to the project you just created. To do this, modify the dependencies section of the package.json file of your CDK project to include the following line:

"@aws-cdk/cloudformation-include": "^1.63.0"

Run either npm install or yarn install, depending on which package manager for Node you’re using.

Now, you need to obtain the template of the MigrationStack that you deployed earlier through the wizard. There are two ways to get it. Either:

  1. Copy CloudFormation JSON block above, and save it as a file on your local machine; or
  2. In the AWS Console for CloudFormation, go into the “Template” tab of the MigrationStack, and copy the contents of the template to your clipboard, and then save it to a file on your local machine.

Whichever method you choose, make sure the saved file is called migration-template.json. Place this new file next to the cdk.json file that is already present in your project. This new file should be placed under version control, just like the rest of your CDK source code.

This is a one-time operation; after the migration, you won’t have to edit the template anymore. All subsequent changes to your stack will happen through CDK code.

To include this template in your CDK application, open the lib/migration-stack.ts file and create an instance of the CfnInclude class just below the comment that says “// The code that defines your stack goes here“. Your code will look like the following example:

import * as cdk from '@aws-cdk/core';
import * as cfn_inc from '@aws-cdk/cloudformation-include';

export class MigrationStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
    const cfnInclude = new cfn_inc.CfnInclude(this, 'Template', { 
      templateFile: 'migration-template.json',
    });
  }
}

Make sure the bin/migration.ts file has the correct name of the Stack to migrate. It should look like this:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
import { MigrationStack } from '../lib/migration-stack';

const app = new cdk.App();
//'MigrationStack' below needs to be the same name as the CloudFormation name
new MigrationStack(app, 'MigrationStack');

After making these code changes, run the cdk diff command in a terminal that has the AWS credentials for the account to which you deployed the MigrationStack. You should see output similar to:

The only difference between the stack running in your account and the stack modeled in your CDK code is related to a CloudFormation Condition object that is used for a CDK-internal resource; but notice that the MyBucket resource from the MigrationStack was included in your CDK application, and the diff command does not show any differences related to it.

Finally, run the cdk deploy command, let the deployment finish, and then run cdk diff again, it should report:

There were no differences.

Congratulations, you have now migrated your CloudFormation template to CDK code! That was easy, wasn’t it? 🙂

Of course, simply migrating your Stack unchanged is probably not the only thing you want to do with your application. You probably want to change and add some things to that Stack! The following section shows you how to do that.

Modifying resources from the template

Using the cloudformation-include module, you can modify any resource from the included template by referring to it using its Logical ID. As an example, since you know that the MigrationStack template defines an S3 Bucket with the logical ID MyBucket, you can retrieve a reference to that resource using the getResource() method of the CfnInclude class.

const cfnBucket = cfnInclude.getResource('MyBucket');

The getResource() method returns the type CfnResource; however, if you know the underlying CDK class that corresponds to the given resource type, you can cast the result to the correct type. For instance, you know that the resource type AWS::S3::Bucket corresponds to the CfnBucket class in the @aws-cdk/aws-s3 module; so, you can cast the object returned by getResource() to it:

import * as s3 from '@aws-cdk/aws-s3';

// ...

    const cfnBucket = cfnInclude.getResource('MyBucket') as s3.CfnBucket;
    // cfnBucket is now of type s3.CfnBucket

After you have a reference to a resource, you can modify any of its properties. For instance, you can change the configuration to block public access to the bucket:

cfnBucket.publicAccessBlockConfiguration = {
  blockPublicAcls: true,
};

Now, run cdk diff. You will see output similar to:

$ cdk diff

Stack MigrationStack
Resources
[~] AWS::S3::Bucket Template/MyBucket MyBucket 
└─ [+] PublicAccessBlockConfiguration
└─ {"BlockPublicAcls":true}

If you were to run the cdk deploy command, the PublicAccessBlockConfiguration property of MyBucket would be changed in MigrationStack.

In this manner, you can modify any property of any resource that is defined in your imported template, directly from CDK code, using features like auto-completion and type checking of your programming language. You no longer have to edit JSON or YAML files to modify your template!

Referencing resources from your template

Similarly, you can reference any resource from the included template when creating new resources.

To see how this works, you will now create a new IAM Role and grant it read access to the MyBucket resource that you referenced above:

import * as cdk from '@aws-cdk/core';
import * as cfn_inc from '@aws-cdk/cloudformation-include';
import * as s3 from '@aws-cdk/aws-s3';
import * as iam from '@aws-cdk/aws-iam';

export class MigrationStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
    const cfnInclude = new cfn_inc.CfnInclude(this, 'Template', {
      templateFile: 'migration-template.json',
    });
    const cfnBucket = cfnInclude.getResource('MyBucket') as s3.CfnBucket;

    const role = new iam.Role(this, 'Role', {
      assumedBy: new iam.AccountRootPrincipal(),
    });
    role.addToPolicy(new iam.PolicyStatement({
      actions: [
        's3:GetObject*',
        's3:GetBucket*',
        's3:List*',
      ],
      resources: [cfnBucket.attrArn],
    }));
  }
}

When you run cdk diff, you should see the new resources added, while referencing the MyBucket resource that is defined in the CloudFormation template:

Creating higher-level resources from the CFN resources

In the previous section, you granted read access to the MyBucket resource to an IAM Role by creating a Policy that explicitly listed all actions that Role is allowed to perform. This required intimate knowledge of S3 and IAM to know exactly which permissions needed to be granted. In this section, you will achieve the same result, but using the CDK Construct Library. The Construct Library allows expressing complex infrastructure with very little code, leveraging higher-level abstractions which, in this case, contain convenient grant* methods that generate least-permission, tightly scoped IAM policies for you. For example, the Construct Library for S3 has the Bucket class, which is a higher-level abstraction over the CfnBucket class.

You can use the from* static methods of the resources in the Construct Library to get a higher-level class from the lower-level Cfn* instance. For example, for Buckets, you use the fromBucketName method of the Bucket class:

const bucket = s3.Bucket.fromBucketName(this, 'Bucket', cfnBucket.ref);

Now with a Bucket instance, you can use the full power of the Construct Library abstractions; for example, you can use the grantRead method to allow our new Role to read from MyBucket instead of hard-coding the exact required S3 permissions:

import * as cdk from '@aws-cdk/core';
import * as cfn_inc from '@aws-cdk/cloudformation-include';
import * as s3 from '@aws-cdk/aws-s3';
import * as iam from '@aws-cdk/aws-iam';

export class MigrationStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
    const cfnInclude = new cfn_inc.CfnInclude(this, 'Template', {
      templateFile: 'migration-template.json',
    });
    const cfnBucket = cfnInclude.getResource('MyBucket') as s3.CfnBucket;
    const bucket = s3.Bucket.fromBucketName(this, 'Bucket', cfnBucket.ref);

    const role = new iam.Role(this, 'Role', {
      assumedBy: new iam.AccountRootPrincipal(),
    });
    bucket.grantRead(role);
  }
}

If you run the cdk diff command with the above code, the output will be the same as when you explicitly granted the Role permissions to cfnBucket.

Cleaning up

To clean up the MigrationStack that you deployed while following the directions in this post, run the following command in your CDK project’s root directory:

$ cdk destroy

This will (after prompting if you are sure) delete the CloudFormation template, and all resources that are inside it, from your AWS account.

Summary

The cloudformation-include module allows you to migrate to CDK from CloudFormation, without many of the shortcomings of the current CfnInclude class from the core package. Resources can be directly imported from your template and modified, even across nested stacks, in templates specified in YAML or JSON formats. All sections of the template, including Resources, Parameters (which you can provide values for when including your template), and Outputs are fully supported and can be directly referenced in your CDK code. This makes migrating your infrastructure from CloudFormation to CDK quick and easy, and you can continue to modify it leveraging the full power of a programming language, and CDK’s higher-level abstractions.

Try migrating an existing stack! Once the migration is complete, you no longer need to modify the CloudFormation template; the CDK application is the source of truth. All changes to your stack can be made through CDK code alone.

The ability to migrate existing CloudFormation templates and stacks to CDK code is available today as developer preview. Give it a try with your stacks and let us know what you think on GitHub!

 

Tirumarai Selvan

Calvin Combs

Shout out to Calvin Combs, summer intern on the CDK team who helped develop the CloudFormation Include construct library. Calvin is a Computer Science and Math double major at University of Maryland, College Park