How can I create an Amazon S3 notification configuration for Lambda on an existing Amazon S3 bucket using AWS CloudFormation?

Last updated: 2019-07-17

How can I use an existing Amazon Simple Storage Service (Amazon S3) bucket to create an Amazon S3 notification configuration for an AWS Lambda function using AWS CloudFormation?

Short Description

To create an Amazon S3 notification configuration, you can use AWS CloudFormation to create a new S3 bucket. Then, add a notification configuration to that bucket using the NotificationConfiguration property. Or, manually add a notification configuration to an existing S3 bucket.

The following steps show you how to add a notification configuration to your existing S3 bucket with AWS CloudFormation by using a Lambda-backed custom resource created in Python 3.6. You'll use a custom resource to trigger a Lambda function, which triggers the PutBucketNotification API to add a notification configuration to your S3 bucket.

Resolution

1. Create a Python file named LambdaS3.py that includes the following code:

from __future__ import print_function
import json
import boto3
import urllib
from botocore.vendored import requests

SUCCESS = "SUCCESS"
FAILED = "FAILED"

print('Loading function')
s3 = boto3.resource('s3')

def lambda_handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))
    responseData={}
    try:
        if event['RequestType'] == 'Delete':
            print("Request Type:",event['RequestType'])
            Bucket=event['ResourceProperties']['Bucket']
            delete_notification(Bucket)
            print("Sending response to custom resource after Delete")
        elif event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
            print("Request Type:",event['RequestType'])
            LambdaArn=event['ResourceProperties']['LambdaArn']
            Bucket=event['ResourceProperties']['Bucket']
            add_notification(LambdaArn, Bucket)
            responseData={'Bucket':Bucket}
            print("Sending response to custom resource")
        responseStatus = 'SUCCESS'
    except Exception as e:
        print('Failed to process:', e)
        responseStatus = 'FAILURE'
        responseData = {'Failure': 'Something bad happened.'}
    send(event, context, responseStatus, responseData)

def add_notification(LambdaArn, Bucket):
    bucket_notification = s3.BucketNotification(Bucket)
    response = bucket_notification.put(
      NotificationConfiguration={
        'LambdaFunctionConfigurations': [
            {
                'LambdaFunctionArn': LambdaArn,
                'Events': [
                    's3:ObjectCreated:*'
                ]
            }
        ]
      }
    )
    print("Put request completed....")

def delete_notification(Bucket):
    bucket_notification = s3.BucketNotification(Bucket)
    response = bucket_notification.put(
      NotificationConfiguration={}
    )
    print("Delete request completed....")


def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False):
    responseUrl = event['ResponseURL']
    print(responseUrl)
    responseBody = {'Status': responseStatus,
                    'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,
                    'PhysicalResourceId': physicalResourceId or context.log_stream_name,
                    'StackId': event['StackId'],
                    'RequestId': event['RequestId'],
                    'LogicalResourceId': event['LogicalResourceId'],
                    'Data': responseData}
    json_responseBody = json.dumps(responseBody)
    print("Response body:\n" + json_responseBody)
    headers = {
        'content-type' : '',
        'content-length' : str(len(json_responseBody))
    }
    try:
        response = requests.put(responseUrl,
                                data=json_responseBody,
                                headers=headers)
        print("Status code: " + response.reason)
    except Exception as e:
        print("send(..) failed executing requests.put(..): " + str(e))

2. Create a zip file named LambdaS3.zip for the Lambda function. See the following example:

-> LambdaS3.zip | |-> LambdaS3.py

Important: Be sure that LambdaS3.py is located at the root level of the zip file.

3. Upload the zip file to an S3 bucket that's in the same AWS Region as your AWS CloudFormation stack. To upload the zip file using the AWS Command Line Interface (AWS CLI), run the following command from the folder containing the LambdaS3.zip file:

aws s3 cp ./LambdaS3.zip s3://awsexamplebucket1/LambdaS3.zip

4. Create an AWS CloudFormation template called LambdaS3.template that includes the following code:

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  Sample template to illustrate use of existing S3 bucket as event source for
  Lambda function
Parameters:
  NotificationBucket:
    Type: String
    Description: S3 bucket which is used for Lambda event notification
  LambdaCodeBucket:
    Type: String
    Description: S3 bucket in which custom lambda code is stored
  LambdaCodeKey:
    Type: String
    Description: Zip file name in which custom lambda code is stored
    Default: LambdaS3.zip
  Lambdahandler:
    Type: String
    Description: Python file name which is packed inside the zip file
    Default: LambdaS3

Resources:
  MainLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        ZipFile: !Join
          - |+

          - - import json
            - 'def lambda_handler(event,context):'
            - '    return ''Welcome... This is a test Lambda Function'''
      Handler: index.lambda_handler
      Role: !GetAtt LambdaIAMRole.Arn
      Runtime: python3.6
      Timeout: 5

  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt MainLambdaFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: s3.amazonaws.com
      SourceAccount: !Ref 'AWS::AccountId'
      SourceArn: !Sub 'arn:aws:s3:::${NotificationBucket}'

  LambdaIAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'

  CustomLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Handler: !Sub '${Lambdahandler}.lambda_handler'
      Role: !GetAtt LambdaIAMRole.Arn
      Code:
        S3Bucket: !Ref LambdaCodeBucket
        S3Key: !Ref LambdaCodeKey
      Runtime: python3.6
      Timeout: 50

  Lambdatrigger:
    Type: 'Custom::RouteTableLambda'
    DependsOn: LambdaInvokePermission
    Properties:
      ServiceToken: !GetAtt CustomLambdaFunction.Arn
      LambdaArn: !GetAtt MainLambdaFunction.Arn
      Bucket: !Ref NotificationBucket

Important: In the preceding example, the MainLambdaFunction resource is the Lambda function that the S3 notification configuration is added to. CustomLambdaFunction is the Lambda function used to add the S3 notification configuration.

5. To launch an AWS CloudFormation stack with the LambdaS3.template file, use the AWS CloudFormation console or the following AWS CLI command:

aws cloudformation create-stack --stack-name lambda-s3-notification --template-body file://LambdaS3.template --parameters ParameterKey=NotificationBucket,ParameterValue=existing-bucket-for-lambda-notification ParameterKey=LambdaCodeBucket,ParameterValue=awsexamplebucket1 ParameterKey=Lambdahandler,ParameterValue=LambdaS3 ParameterKey=LambdaCodeKey,ParameterValue=LambdaS3.zip --capabilities CAPABILITY_NAMED_IAM --region us-east-1

Important: When you launch your AWS CloudFormation stack, you must pass in the following:

Your S3 bucket (existing-bucket-for-lambda-notification)
The S3 bucket name (awsexamplebucket1) where you uploaded the zip file
The zip file name (LambdaS3.zip)
The name of the file where you created the Lambda function (LambdaS3.py)

The stack creates a Lambda function and Lambda permissions for Amazon S3. Now, you can use your S3 bucket for Lambda notifications, because the stack added the required notification configuration to your S3 bucket.

Did this article help you?

Yes

Anything we could improve?

Let us know

Need more help?

Contact AWS Support