Resolving circular dependency in provisioning of Amazon S3 buckets with AWS Lambda event notifications

Overview

AWS CloudFormation provides a common language for you to describe and provision all of the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all AWS Regions and accounts.

It is an AWS CloudFormation best practice to have resource names automatically generated because that makes the CloudFormation template easy to test and deploy in multiple Regions and environments. For example, Amazon S3 buckets must have globally unique names. If you hard-code bucket name in your CloudFormation template, that makes the code brittle and not easy to deploy or test. There is a possibility that another customer in the world might have coincidentally chosen to create an S3 bucket with the same name. Thus, wherever possible, we recommend that you do not explicitly name your resources. Instead, let CloudFormation generate the names for you to avoid collisions and indeterministic failures.

At the same time, it is very common to create Amazon S3 buckets with event notifications backed by AWS Lambda functions. When customers attempt to deploy this setup in CloudFormation, it results in a deployment failure due to a circular dependency between the Lambda function and the S3 bucket. This is because Lambda permissions need to know the bucket’s unique Amazon Resource Name (ARN) before you can create the Lambda function, and the bucket needs to know the Lambda function’s unique ARN before you can create the bucket.

In this article, I present a mechanism to resolve the circular dependency while preserving the desired outcome of auto-generated S3 bucket names using CloudFormation custom resources, which enable you to write custom provisioning logic in Lambda functions associated with your templates.

Circular dependency demonstration

Consider the following template that sets up an S3 bucket with events that trigger a Lambda function that simply logs the event.

AWSTemplateFormatVersion: '2010-09-09'
Description: Sample template that demonstrates the circular dependency
Resources:
  TestFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Path: /

  TestEventFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Dummy function, simply logs received S3 events
      Handler: index.handler
      Runtime: python2.7
      Role: !GetAtt 'TestFunctionRole.Arn'
      Timeout: 240
      Code:
        ZipFile: |
          import json
          import logging
          logger = logging.getLogger()
          logger.setLevel(logging.DEBUG)

          def handler(event, context):
              logger.info('Received event: %s' % json.dumps(event))

  TestS3BucketEventPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      SourceAccount: !Ref 'AWS::AccountId'
      FunctionName: !Ref 'TestEventFunction'
      SourceArn: !GetAtt
        - TestS3Bucket
        - Arn
      Principal: s3.amazonaws.com

  TestS3Bucket:
    Type: AWS::S3::Bucket
    DependsOn: TestS3BucketEventPermission
    Properties:
      NotificationConfiguration:
        LambdaConfigurations:
          -
            Function:
              "Fn::GetAtt":
                - 'TestEventFunction'
                - 'Arn'
            Event: s3:ObjectCreated:*

When you attempt to launch this template, you receive the following error:

Circular dependency between resources: [TestS3BucketEventPermission, TestS3Bucket].

This is because TestS3BucketEventPermission needs to know the ARN of the TestS3Bucket before it can create TestEventFunction. However, the TestS3Bucket’s NotificationConfiguration needs to know the ARN of TestEventFunction before it can create the TestS3Bucket, thus creating a circular dependency.

If you change the DependsOn property in the TestS3Bucket to point to the TestEventFunction instead, you run into a different error: “Unable to validate the following destination configurations.”

Solution using custom resource

We can avoid the circular dependency demonstrated earlier by creating the S3 bucket without any notification configuration. Then we set up the notification after the Lambda function, Lambda permission, and S3 bucket resources have been created. This can be done manually using the CLI, but CloudFormation custom resources provide a good mechanism to include the logic as part of your automated deployment lifecycle in CloudFormation itself.

In the following example, I create a new Lambda function, ApplyBucketNotificationFunction, that takes an S3 bucket and a Lambda function ARN as event inputs and applies NotificationConfiguration to the S3 bucket with the provided Lambda function.

Note: I have included the code inline for demonstration. You could source the Lambda code zip file from S3 to enable better code maintainability.

AWSTemplateFormatVersion: '2010-09-09'
Description: Sample template that demonstrates setting up Lambda event notification
  for an S3 bucket using a custom resource

Resources:
  TestFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Path: /

  TestEventFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Dummy function, simply logs received S3 events
      Handler: index.handler
      Runtime: python2.7
      Role: !GetAtt 'TestFunctionRole.Arn'
      Timeout: 240
      Code:
        ZipFile: |
          import json
          import logging
          logger = logging.getLogger()
          logger.setLevel(logging.DEBUG)

          def handler(event, context):
              logger.info('Received event: %s' % json.dumps(event))

  TestS3BucketEventPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      SourceAccount: !Ref 'AWS::AccountId'
      FunctionName: !Ref 'TestEventFunction'
      SourceArn: !GetAtt
        - TestS3Bucket
        - Arn
      Principal: s3.amazonaws.com

  TestS3Bucket:
    Type: AWS::S3::Bucket
    DependsOn: TestEventFunction

  ApplyNotificationFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Path: /
      Policies:
        - PolicyName: S3BucketNotificationPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowBucketNotification
                Effect: Allow
                Action: s3:PutBucketNotification
                Resource:
                  - !Sub 'arn:aws:s3:::${TestS3Bucket}'
                  - !Sub 'arn:aws:s3:::${TestS3Bucket}/*'

  ApplyBucketNotificationFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Dummy function, just logs the received event
      Handler: index.handler
      Runtime: python2.7
      Role: !GetAtt 'ApplyNotificationFunctionRole.Arn'
      Timeout: 240
      Code:
        ZipFile: |
          import boto3
          import logging
          import json
          import cfnresponse

          s3Client = boto3.client('s3')
          logger = logging.getLogger()
          logger.setLevel(logging.DEBUG)

          def addBucketNotification(bucketName, notificationId, functionArn):
            notificationResponse = s3Client.put_bucket_notification_configuration(
              Bucket=bucketName,
              NotificationConfiguration={
                'LambdaFunctionConfigurations': [
                  {
                    'Id': notificationId,
                    'LambdaFunctionArn': functionArn,
                    'Events': [
                      's3:ObjectCreated:*'
                    ]
                  },
                ]
              }
            )
            return notificationResponse

          def create(properties, physical_id):
            bucketName = properties['S3Bucket']
            notificationId = properties['NotificationId']
            functionArn = properties['FunctionARN']
            response = addBucketNotification(bucketName, notificationId, functionArn)
            logger.info('AddBucketNotification response: %s' % json.dumps(response))
            return cfnresponse.SUCCESS, physical_id

          def update(properties, physical_id):
            return cfnresponse.SUCCESS, None

          def delete(properties, physical_id):
            return cfnresponse.SUCCESS, None

          def handler(event, context):
            logger.info('Received event: %s' % json.dumps(event))

            status = cfnresponse.FAILED
            new_physical_id = None

            try:
              properties = event.get('ResourceProperties')
              physical_id = event.get('PhysicalResourceId')

              status, new_physical_id = {
                'Create': create,
                'Update': update,
                'Delete': delete
              }.get(event['RequestType'], lambda x, y: (cfnresponse.FAILED, None))(properties, physical_id)
            except Exception as e:
              logger.error('Exception: %s' % e)
              status = cfnresponse.FAILED
            finally:
              cfnresponse.send(event, context, status, {}, new_physical_id)

  ApplyNotification:
    Type: Custom::ApplyNotification
    Properties:
      ServiceToken: !GetAtt 'ApplyBucketNotificationFunction.Arn'
      S3Bucket: !Ref 'TestS3Bucket'
      FunctionARN: !GetAtt 'TestEventFunction.Arn'
      NotificationId: S3ObjectCreatedEvent

Conclusion

At the beginning of this blog post we saw that the Lambda permission (TestS3BucketEventPermission) and the NotificationConfiguration of the S3 bucket (TestS3Bucket) resources were dependent on each other, thus CloudFormation couldn’t create either resource.

I provided a solution example that avoids this circular dependency. I created the TestS3Bucket first without notification configurations. After that, I used a custom resource ApplyNotification that invokes the ApplyBucketNotificationFunction, which adds the Lambda notification configuration to the S3 bucket after both TestS3Bucket and TestEventFunction have been created.

There are two advantages to this solution:

First, S3 bucket names have to globally unique, so any solution that requires hardcoding bucket names runs the risk of indeterminate deployment errors in the event of a name collision. This approach prevents that scenario because it does not require S3 buckets to be given a fixed name.

Second, by using the custom resource I have made applying the bucket notification a part of the CloudFormation deployment. This eliminates any need for manual or scripting work necessary to set up the bucket notification after the stack has successfully deployed.

References

You can access code for the two templates below.

About the author

Vinod Shukla is a Partner Solutions Architect at Amazon Web Services. As part of the AWS Quick Starts team, he enjoys working with partners providing technical guidance and assistance in building gold-standard reference deployments.

AWS Cloud Operations & Migrations Blog