How do I use custom resources with Amazon S3 buckets in AWS CloudFormation?

Last updated: 2020-08-19

I want to use custom resources with Amazon Simple Storage Service (Amazon S3) buckets in AWS CloudFormation, so that I can perform standard operations after creating an S3 bucket.

Short description

You can use the AWS CloudFormation template in the following resolution to use custom resources with an S3 bucket in AWS CloudFormation.

Consider the following:

  • The template allows you to create folders in S3 buckets. Amazon S3 has a flat structure, but supports the folder concept as a means of grouping objects.
  • You can use the template to perform operations after creating an S3 bucket, including copying content, uploading content, and synchronizing two different buckets.
  • You can modify the template with your own code.
  • This template uses AWS Lambda-backed custom resources, and assumes that you're familiar with Lambda best practices and troubleshooting issues.

Note: In the following resolution, all the S3 bucket content is deleted when the AWS CloudFormation stack is deleted. You can modify this behavior by modifying the Lambda code.

Resolution

Get the AWS CloudFormation template

To create folders in an S3 bucket using AWS CloudFormation, save the following AWS CloudFormation template as a YAML file:

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this
software and associated documentation files (the "Software"), to deal in the Software
without restriction, including without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

AWSTemplateFormatVersion: 2010-09-09
Description: Working with custom resources and S3
Parameters:
  S3BucketName:
    Type: String
    Description: "S3 bucket to create."
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9_-]*"
  DirsToCreate:
    Description: "Comma delimited list of directories to create."
    Type: CommaDelimitedList
Resources:
  SampleS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref S3BucketName
  S3CustomResource:
    Type: Custom::S3CustomResource
    Properties:
      ServiceToken: !GetAtt AWSLambdaFunction.Arn
      the_bucket: !Ref SampleS3Bucket
      dirs_to_create: !Ref DirsToCreate
  AWSLambdaFunction:
     Type: "AWS::Lambda::Function"
     Properties:
       Description: "Work with S3 Buckets!"
       FunctionName: !Sub '${AWS::StackName}-${AWS::Region}-lambda'
       Handler: index.handler
       Role: !GetAtt AWSLambdaExecutionRole.Arn
       Timeout: 360
       Runtime: python3.6
       Code:
         ZipFile: |
          import boto3
          import cfnresponse
          def handler(event, context):
              # Init ...
              the_event = event['RequestType']
              print("The event is: ", str(the_event))
              response_data = {}
              s_3 = boto3.client('s3')
              # Retrieve parameters
              the_bucket = event['ResourceProperties']['the_bucket']
              dirs_to_create = event['ResourceProperties']['dirs_to_create']
              try:
                  if the_event in ('Create', 'Update'):
                      print("Requested folders: ", str(dirs_to_create))
                      for dir_name in dirs_to_create:
                          print("Creating: ", str(dir_name))
                          s_3.put_object(Bucket=the_bucket,
                                         Key=(dir_name
                                              + '/'))
                  elif the_event == 'Delete':
                      print("Deleting S3 content...")
                      b_operator = boto3.resource('s3')
                      b_operator.Bucket(str(the_bucket)).objects.all().delete()
                  # Everything OK... send the signal back
                  print("Execution succesfull!")
                  cfnresponse.send(event,
                                   context,
                                   cfnresponse.SUCCESS,
                                   response_data)
              except Exception as e:
                  print("Execution failed...")
                  print(str(e))
                  response_data['Data'] = str(e)
                  cfnresponse.send(event,
                                   context,
                                   cfnresponse.FAILED,
                                   response_data)
  AWSLambdaExecutionRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
         Statement:
         - Action:
           - sts:AssumeRole
           Effect: Allow
           Principal:
             Service:
             - lambda.amazonaws.com
         Version: '2012-10-17'
       Path: "/"
       Policies:
       - PolicyDocument:
           Statement:
           - Action:
             - logs:CreateLogGroup
             - logs:CreateLogStream
             - logs:PutLogEvents
             Effect: Allow
             Resource: arn:aws:logs:*:*:*
           Version: '2012-10-17'
         PolicyName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambda-CW
       - PolicyDocument:
           Statement:
           - Action:
             - s3:PutObject
             - s3:DeleteObject
             - s3:List*
             Effect: Allow
             Resource:
             - !Sub arn:aws:s3:::${SampleS3Bucket}/*
             - !Sub arn:aws:s3:::${SampleS3Bucket}
           Version: '2012-10-17'
         PolicyName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambda-S3
       RoleName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambdaExecutionRole

Deploy your AWS CloudFormation template

You can deploy your AWS CloudFormation template using either the AWS CloudFormation console or the AWS Command Line Interface (AWS CLI).

Using the AWS CloudFormation console:

1.    Open the AWS CloudFormation console.

2.    Choose Create stack, and then choose With new resources (standard).

3.    In the Specify template section, choose Upload a template file.

4.    Choose Choose file, select the template that you downloaded in step 1, and then choose Next.

5.    In the Parameters section, for S3BucketName, choose your S3 bucket.

6.    For DirsToCreate, enter a comma-delimited list of folders and subfolders that you want to create.

Note: For example, you can enter dir_1,dir_2/sub_dir_2,dir_3 as a list.

7.    Complete the rest of the steps in the setup wizard, and then choose Create stack.

Using the AWS CLI:

1.    Name your downloaded template custom-resource-lmabda-s3.yaml.

2.    Install the AWS CLI.

3.    Open a command line in your operating system, and then go to the folder where the template is located.

4.    Run the following command:

aws cloudformation create-stack \
                   --timeout-in-minutes 10 \
                   --disable-rollback \
                   --stack-name testing-custom-resource-s3 \
                   --template-body file://custom-resource-lmabda-s3.yaml \
                   --capabilities CAPABILITY_NAMED_IAM \
                   --parameters \
                   ParameterKey=DirsToCreate,ParameterValue="dir_1\,dir_2/sub_dir_2\,dir_3" \
                   ParameterKey=S3BucketName,ParameterValue="test-bucket-custom-resource"

Node: Update parameters with your values.


Did this article help?


Do you need billing or technical support?