Amazon S3 authenticated bootstrapping in AWS CloudFormation

A common way to pull files from Amazon S3 is by using utilities like wget and curl from within the Amazon EC2 instance UserData. When you use these file-retrieval tools, however, the S3 bucket must be publicly accessible, which might be undesirable because the contents of the bucket will also be publicly available. To remedy this, you can configure AWS CloudFormation so that all S3 requests are authenticated via AWS Identity and Access Management (IAM) roles associated with an instance.

Solution overview

In this post, I discuss some bootstrapping best practices while performing typical UserData actions, such as retrieving files from an Amazon S3 bucket. To pull source files during UserData bootstrapping, you’ll retrieve objects from a private bucket using cfn-init and an accompanying IAM instance role. You’ll also provide a fully functional template and discuss the components of the code that are needed for authenticated S3 calls.

1. Create an instance role

Start by creating a RootRole that allows the sts:AssumeRole action to run. Then, add a policy to this role that allows the s3:GetObject action to limit access to both the bucket and key prefix where the AWS CloudFormation template is staged.

See RootRole on GitHub.

2. Associate the instance profile with the instance role

In the Roles list of the InstanceProfile, reference the RootRole.

See RootInstanceProfile on GitHub.

3. Create an example Amazon EC2 instance

Now that you have set the required IAM roles to access the Amazon S3 objects, create an example Amazon EC2 instance and associate it with RootInstanceProfile.

Note     Add ImageId (via Amazon Machine Image mapping), InstanceType, KeyName, SecurityGroupIds, and SubnetId to your Amazon EC2 instance definitions.

See ExampleInstance on GitHub.

4. Create the Amazon EC2 instance metadata

In the EC2 instance metadata, create AWS::CloudFormation::Authentication. Set the bucket toBucketName, and set roleName to RootRole.

See Metadata (AWS::CloudFormation::Authentication) on GitHub.

5. Build the cfn-init metadata

For this example, I create configSets that calls a sample called helloworld_cfg. This part of the script resides in the same S3 bucket as our test template, but in a separate folder: BucketName/KeyPrefix/scripts/print_stdout.sh.

See configSet on GitHub.

6. Invoke cfn-init from UserData and signal it upon completion

While bootstrapping this instance, I use quickstart-linux-utilities to prepare the UserData section. For more information, see the Introduction to quickstart-linux-utilities blog post.

See cfn-init on GitHub.

For the complete example, see Quick Start’s cfn-authenticated-s3-example code on GitHub.

Conclusion

When developing AWS CloudFormation templates that require access to Amazon S3 for bootstrapping, this pattern provides an authenticated and natively supported method for accessing your S3 files without the need for public access.

For additional examples, see our Quick Start references that use secure S3 authentication: Corda Blockchain and HashiCorp Consul.