Integration & Automation
Amazon S3 authenticated bootstrapping in AWS CloudFormation
A common way to pull files from Amazon S3 is by using utilities like
curl from within the Amazon EC2 instance
UserData. When you use these file-retrieval tools, however, the S3 bucket must be publicly accessible, which might be undesirable because the contents of the bucket will also be publicly available. To remedy this, you can configure AWS CloudFormation so that all S3 requests are authenticated via AWS Identity and Access Management (IAM) roles associated with an instance.
In this post, I discuss some bootstrapping best practices while performing typical
UserData actions, such as retrieving files from an Amazon S3 bucket. To pull source files during
UserData bootstrapping, you’ll retrieve objects from a private bucket using
cfn-init and an accompanying IAM instance role. You’ll also provide a fully functional template and discuss the components of the code that are needed for authenticated S3 calls.
1. Create an instance role
Start by creating a
RootRole that allows the
sts:AssumeRole action to run. Then, add a policy to this role that allows the
s3:GetObject action to limit access to both the bucket and key prefix where the AWS CloudFormation template is staged.
See RootRole on GitHub.
2. Associate the instance profile with the instance role
Roles list of the
InstanceProfile, reference the
See RootInstanceProfile on GitHub.
3. Create an example Amazon EC2 instance
Now that you have set the required IAM roles to access the Amazon S3 objects, create an example Amazon EC2 instance and associate it with
ImageId (via Amazon Machine Image mapping),
SubnetId to your Amazon EC2 instance definitions.
See ExampleInstance on GitHub.
4. Create the Amazon EC2 instance metadata
In the EC2 instance metadata, create
AWS::CloudFormation::Authentication. Set the bucket to
BucketName, and set
See Metadata (AWS::CloudFormation::Authentication) on GitHub.
5. Build the cfn-init metadata
For this example, I create
configSets that calls a sample called
helloworld_cfg. This part of the script resides in the same S3 bucket as our test template, but in a separate folder:
See configSet on GitHub.
6. Invoke cfn-init from UserData and signal it upon completion
While bootstrapping this instance, I use
quickstart-linux-utilities to prepare the
UserData section. For more information, see the Introduction to quickstart-linux-utilities blog post.
See cfn-init on GitHub.
For the complete example, see Quick Start’s cfn-authenticated-s3-example code on GitHub.
When developing AWS CloudFormation templates that require access to Amazon S3 for bootstrapping, this pattern provides an authenticated and natively supported method for accessing your S3 files without the need for public access.
For additional examples, see our Quick Start references that use secure S3 authentication: Corda Blockchain and HashiCorp Consul.