AWS Cloud Operations & Migrations Blog
Introducing AWS CloudFormation modules
If you’ve used AWS CloudFormation, you’ve probably experienced times when you are trying to build applications and want to deploy resources with best practices defined. As you work on your templates, you might be curious about which resource properties to configure and which values to use to follow those best practices. While you’re building your application, you might want to just follow the best practices for a resource and not worry about all the properties and their possible values.
To help solve this issue, the CloudFormation team is excited to announce the release of modules. Modules are building blocks that can be reused across multiple CloudFormation templates and is used just like a native CloudFormation resource.
These building blocks can be for a single resource, like best practices for defining an Amazon Elastic Compute Cloud (Amazon EC2) instance or they can be for multiple resources, to define common patterns of application architecture. These building blocks can be nested into other modules, so you can stack your best practices into higher-level building blocks. This means you can create a module that defines your organization’s standards for a Lambda function and then consume that Lambda module in another module that defines the patterns for your serverless Amazon API Gateway implementation.
CloudFormation modules are available in the CloudFormation registry, so you can use them just like a native resource. A module with a resource type is postfixed in the CloudFormation registry with ::MODULE so it’s easy to denote when you are using a module or a native registry resource. Parameters that are defined in the CloudFormation module become properties when consuming the ::MODULE resource type. When you use a CloudFormation module, the module template is expanded into the consuming template, which makes it possible for you to access the resources inside the module using a Ref or Fn::GetAtt.
Getting started
There are two ways to create your CloudFormation modules:
- You can use the resource types,
AWS::CloudFormation::ModuleVersionandAWS::CloudFormation::ModuleDefaultVersion, in a CloudFormation template. - You can use the CloudFormation Command Line Interface (CLI). This is the recommended method because it offers a guided development process. To install the CloudFormation CLI, follow these instructions.
Create your first module
A typical application requires an Amazon Simple Storage Service (Amazon S3) bucket. The bucket has many configurable settings, including encryption, public access block configurations, and access control. In this post, you create a restrictive bucket policy that limits access to the bucket and makes sure traffic to the bucket is using https. As you evaluate all of your organization’s standards, you start to realize the number of things that need to be configured and repeated every time you want an S3 bucket. In this example, you build this best practice S3 bucket module once and reuse it repeatedly, without any additional work. Then you can use this module to provision other AWS services that use the bucket created in the module.
First, create an empty directory to store the module.
>> mkdir s3-module && cd s3-moduleThen, initialize the folder to hold the module. When you run init, you can now pick between a resource or a module. When you choose a name for your module, be sure to use one that isn’t reserved. The name must end with ::MODULE.
>> cfn init
Initializing new project
Do you want to develop a new resource(r) or a module(m)?.
>> m
What's the name of your module type?
(<Organization>::<Service>::<Name>::MODULE)
>> MyCompany::S3::Bucket::MODULE
When this command is complete, it creates the following:
- A
fragmentsfolder that contains the CloudFormation template you are going to use for the module. - An
.rpdk-configfile that contains details about the module, including its name. - An
rpdk.logthat contains logs from runningcfncommands.
CloudFormation modules supports both JSON and YAML templates but for this example we will use JSON. In this example, you delete the default JSON file in the fragments folder and create a new file named s3.json. You can have only one template in the fragments folder, so remove any examples created for you by cfn init.
You create the module and its resources in the s3.json file. In this template, you create an S3 bucket, an AWS Key Management Service (AWS KMS) key to encrypt data at rest inside the S3 bucket, and a bucket policy that restricts access to the S3 bucket to the provided IAM roles and requires encryption when communicating with the S3 bucket. The Parameters section is used as resource properties in the module template. Outputs are merged into the Outputs section of the template using the module.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Create a S3 bucket that follows MyCompany's standards",
"Parameters": {
"KMSKeyAlias": {
"Description": "The alias for your KMS key. If you will leave this field empty KMS key alias won't be created along the key.",
"Type": "String",
"AllowedPattern": "^(?!aws)([a-zA-Z0-9\\-\\_\\/]){1,32}$|^$",
"MinLength": 0,
"MaxLength": 32,
"ConstraintDescription": "KMS key alias must be at least 1 and no more than 32 characters long, can contain lowercase and uppercase letters, numbers, hyphens, underscores and forward slashes and cannot begin with aws."
},
"ReadOnlyArn": {
"Description": "Provide ARN of an existing Principal (role) that will be granted with read only access to the S3 bucket (e.g. 'arn:aws:iam::123456789xxx:role/myS3ROrole'). If not specified, access will be granted to current AWS account:root only. CF deployment will fail and rollback for non-existing ARN.",
"Type": "String",
"Default": "",
"AllowedPattern": "^(arn:aws:iam::\\d{12}:role(\\/|\\/[\\w\\!\\\"\\#\\$\\%\\'\\(\\)\\*\\+\\,\\-\\.\\/\\:\\;\\<\\=\\>\\?\\@\\[\\\\\\]\\^\\`\\{\\|\\}\\~]{1,510}\\/)[\\w\\+\\=\\,\\.\\@\\-]{1,64})$|^$",
"ConstraintDescription": "IAM role ARN must start with arn:aws:iam::<12-digit AWS account number>:role/[<path>/]<name of role>. The name of role must be at least 1 and no more than 64 characters long, can contain lowercase letters, uppercase letters, numbers, plus (+), equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-). Path is optional and must not exceed 510 characters."
},
"ReadWriteArn": {
"Description": "Provide ARN of an existing Principal (role) that will be granted with read and write access to the S3 bucket (e.g. 'arn:aws:iam::123456789xxx:role/myS3RWrole'). If not specified, access will be granted to current AWS account:root only. CF deployment will fail and rollback for non-existing ARN.",
"Type": "String",
"Default": "",
"AllowedPattern": "^(arn:aws:iam::\\d{12}:role(\\/|\\/[\\w\\!\\\"\\#\\$\\%\\'\\(\\)\\*\\+\\,\\-\\.\\/\\:\\;\\<\\=\\>\\?\\@\\[\\\\\\]\\^\\`\\{\\|\\}\\~]{1,510}\\/)[\\w\\+\\=\\,\\.\\@\\-]{1,64})$|^$",
"ConstraintDescription": "IAM role ARN must start with arn:aws:iam::<12-digit AWS account number>:role/[<path>/]<name of role>. The name of role must be at least 1 and no more than 64 characters long, can contain lowercase letters, uppercase letters, numbers, plus (+), equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-). Path is optional and must not exceed 510 characters."
}
},
"Resources": {
"KmsKey": {
"Type": "AWS::KMS::Key",
"DeletionPolicy": "Retain",
"Properties": {
"Enabled": true,
"EnableKeyRotation": true,
"KeyPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Give AWS account:root full control over the KMS key",
"Effect": "Allow",
"Principal": {
"AWS": {
"Fn::Sub": "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
}
},
"Action": [
"kms:*"
],
"Resource": "*"
},
{
"Sid": "Give ReadOnlyRole access to use KMS key for decryption",
"Effect": "Allow",
"Principal": {
"AWS": {
"Ref": "ReadOnlyArn"
}
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Give the ReadWriteRole access to use KMS key for encryption and decryption",
"Effect": "Allow",
"Principal": {
"AWS": {
"Ref": "ReadWriteArn"
}
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
}
},
"KmsKeyAlias": {
"Type": "AWS::KMS::Alias",
"Properties": {
"AliasName": {
"Fn::Sub": "alias/${KMSKeyAlias}"
},
"TargetKeyId": {
"Ref": "KmsKey"
}
}
},
"Bucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"AccessControl": "BucketOwnerFullControl",
"BucketEncryption": {
"ServerSideEncryptionConfiguration": [
{
"ServerSideEncryptionByDefault": {
"KMSMasterKeyID": {
"Ref": "KmsKey"
},
"SSEAlgorithm": "aws:kms"
}
}
]
},
"BucketName": {
"Fn::Sub": "${AWS::StackName}-${AWS::AccountId}-${AWS::Region}"
},
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}
}
},
"BucketPolicy": {
"Type": "AWS::S3::BucketPolicy",
"Properties": {
"Bucket": {
"Ref": "Bucket"
},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
},
"Condition": {
"StringEquals": {
"s3:x-amz-server-side-encryption": "AES256"
}
}
},
{
"Sid": "DenyPublicReadACL",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
},
"Condition": {
"StringEquals": {
"s3:x-amz-acl": [
"public-read",
"public-read-write",
"authenticated-read"
]
}
}
},
{
"Sid": "DenyPublicReadGrant",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
},
"Condition": {
"StringLike": {
"s3:x-amz-grant-read": [
"*http://acs.amazonaws.com/groups/global/AllUsers*",
"*http://acs.amazonaws.com/groups/global/AuthenticatedUsers*"
]
}
}
},
{
"Sid": "DenyNonHttpsConnections",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
},
"Condition": {
"Bool": {
"aws:SecureTransport": false
}
}
},
{
"Sid": "Give ReadOnlyRole access to get objects from bucket and list bucket",
"Effect": "Allow",
"Principal": {
"AWS": {
"Ref": "ReadOnlyArn"
}
},
"Action": [
"s3:GetObject",
"s3:GetObjectTagging",
"s3:ListBucket",
"s3:ListBucketByTags"
],
"Resource": [
{
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}"
},
{
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
}
]
},
{
"Sid": "Give the ReadWriteRole access to get and put objects from and to bucket and list bucket and multipart uploads",
"Effect": "Allow",
"Principal": {
"AWS": {
"Ref": "ReadWriteArn"
}
},
"Action": [
"s3:DeleteObject",
"s3:DeleteObjectTagging",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:ListBucket",
"s3:ListBucketByTags",
"s3:PutObject",
"s3:PutObjectTagging"
],
"Resource": [
{
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}"
},
{
"Fn::Sub": "arn:${AWS::Partition}:s3:::${Bucket}/*"
}
]
}
]
}
}
}
},
"Outputs": {
"BucketArn": {
"Description": "ARN of the bucket created.",
"Value": {
"Fn::GetAtt": [
"Bucket",
"Arn"
]
}
},
"BucketName": {
"Description": "Name of the bucket created.",
"Value": {
"Ref": "Bucket"
}
},
"KmsKeyAlias": {
"Description": "Alias of SSE-KMS Customer Managed Key used to encrypt S3 bucket content.",
"Value": {
"Ref": "KmsKeyAlias"
}
},
"KmsKeyArn": {
"Description": "ARN of SSE-KMS Customer Managed Key used to encrypt S3 bucket content.",
"Value": {
"Fn::GetAtt": [
"KmsKey",
"Arn"
]
}
}
}
}
After you have updated the s3.json file, you can submit it to the CloudFormation registry.
>> cfn submit
Successfully submitted type. Waiting for registration with token '{token}' to complete.
Registration complete.
{token} is a random token for your resource. After the registration is complete, the module is available for your use.
You can view this module from the CloudFormation console or by running the following command:
>> aws cloudformation describe-type --type MODULE --type-name MyCompany::S3::Bucket::MODULEThe describe-type command describes the types available in the registry. Pass in the type parameter to filter the types to modules and then supply the type-name for the module.
The CloudFormation CLI creates a schema file, schema.json, for the template in the root of the directory. If you’re familiar with CloudFormation registry resource types, this schema is critical for your resource to work. With modules, this schema is generated from the provided template automatically. Do not edit the schema.json file.
Use your first module
Because a CloudFormation module is now a resource type, you can use it like any other resource type in your CloudFormation template. In the s3-module folder, create a new CloudFormation template named firehose.yaml. Add the following code to the file.
AWSTemplateFormatVersion: '2010-09-09'
Description: "Create a Firehose stream that writes to S3"
Resources:
FirehoseDestination:
Type: MyCompany::S3::Bucket::MODULE
Properties:
KMSKeyAlias: !Sub "${AWS::StackName}"
ReadWriteArn: !GetAtt FirehoseRole.Arn
ReadOnlyArn: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
FirehoseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AssumeRole1
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
FirehosePolicy:
Type: 'AWS::IAM::Policy'
Properties:
PolicyName: "ReadWrite"
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: "KmsEncryptionDecryption"
Effect: Allow
Action:
- 'kms:Decrypt'
- 'kms:GenerateDataKey'
Resource: !GetAtt FirehoseDestinationKmsKey.Arn
Condition:
StringEquals:
kms:ViaService: !Sub 's3:${AWS::Region}.amazonaws.com'
StringLike:
kms:EncryptionContext:aws:s3:arn: !Sub '${FirehoseDestinationBucket.Arn}/*'
- Sid: FirehoseAccess
Effect: Allow
Action:
- kinesis:DescribeStream
- kinesis:GetShardIterator
- kinesis:GetRecords
- kinesis:ListShards
Resource: !GetAtt Firehose.Arn
- Sid: "S3ListBucket"
Effect: Allow
Action:
- 's3:ListBucket'
- 's3:ListBucketByTags'
- 's3:ListBucketMultipartUploads'
- 's3:GetBucketLocation'
Resource: !GetAtt FirehoseDestinationBucket.Arn
- Sid: "S3GetPutDeleteObject"
Effect: Allow
Action:
- 's3:DeleteObject'
- 's3:DeleteObjectTagging'
- 's3:GetObject'
- 's3:GetObjectTagging'
- 's3:PutObject'
- 's3:PutObjectTagging'
Resource: !Sub '${FirehoseDestinationBucket.Arn}/*'
Roles:
- !Ref FirehoseRole
Firehose:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamName: !Sub "${AWS::StackName}"
DeliveryStreamType: DirectPut
S3DestinationConfiguration:
BucketARN: !GetAtt FirehoseDestinationBucket.Arn
RoleARN: !GetAtt FirehoseRole.Arn
EncryptionConfiguration:
KMSEncryptionConfig:
AWSKMSKeyARN: !GetAtt FirehoseDestinationKmsKey.Arn
The resource FirehoseDestination with Type MyCompany::S3::Bucket::MODULE is the resource that consumes the new module. The properties match the parameters in the module template.
You’ll notice that the Ref and Fn::GetAtt intrinsic functions are accessing resources or parameters that aren’t in this template (!GetAtt FirehoseDestinationKmsKey.Arn). You can access resources in the module by prefixing the logical name of the resource in the module template (for example, KmsKey) with the logical name of the module in the consumer template (for example, FirehoseDestination). So to access the KmsKey ARN in the module, the code uses FirehoseDestinationKmsKey.Arn.
To create a stack with this template:
aws cloudformation create-stack --stack-name s3-blog-module --template-body file://firehose.yaml --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAMSince we are not using a change set, you must specify CAPABILITY_AUTO_EXPAND so the module is expanded when CloudFormation creates the stack.
After the stack is complete, you have an Amazon Kinesis Data Firehose stream that is writing to an S3 bucket. For this stack, the S3 bucket was created using the module. It follows the best practices that were added to the module. As a consumer of this module, you don’t need to copy all those resources and get things configured correctly. The module does the heavy lifting for you.
Conclusion
Your organization should consider using modules to scale its best practices.
We expect the community will provide CloudFormation modules that can be collaborated on through public repositories. AWS has developed some modules to get you started. You can find them on GitHub at aws-cloudformation/aws-cloudformation-samples.
About the author
Editor’s Note: The original post (24 NOV 2020) says that modules only supports JSON. This blog has been updated to include YAML support. (29 JAN 2021)