AWS Cloud Operations Blog
Resolving circular dependency in provisioning of Amazon S3 buckets with AWS Lambda event notifications
Overview
AWS CloudFormation provides a common language for you to describe and provision all of the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all AWS Regions and accounts.
It is an AWS CloudFormation best practice to have resource names automatically generated because that makes the CloudFormation template easy to test and deploy in multiple Regions and environments. For example, Amazon S3 buckets must have globally unique names. If you hard-code bucket name in your CloudFormation template, that makes the code brittle and not easy to deploy or test. There is a possibility that another customer in the world might have coincidentally chosen to create an S3 bucket with the same name. Thus, wherever possible, we recommend that you do not explicitly name your resources. Instead, let CloudFormation generate the names for you to avoid collisions and indeterministic failures.
At the same time, it is very common to create Amazon S3 buckets with event notifications backed by AWS Lambda functions. When customers attempt to deploy this setup in CloudFormation, it results in a deployment failure due to a circular dependency between the Lambda function and the S3 bucket. This is because Lambda permissions need to know the bucket’s unique Amazon Resource Name (ARN) before you can create the Lambda function, and the bucket needs to know the Lambda function’s unique ARN before you can create the bucket.
In this article, I present a mechanism to resolve the circular dependency while preserving the desired outcome of auto-generated S3 bucket names using CloudFormation custom resources, which enable you to write custom provisioning logic in Lambda functions associated with your templates.
Circular dependency demonstration
Consider the following template that sets up an S3 bucket with events that trigger a Lambda function that simply logs the event.
AWSTemplateFormatVersion: '2010-09-09'
Description: Sample template that demonstrates the circular dependency
Resources:
TestFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Path: /
TestEventFunction:
Type: AWS::Lambda::Function
Properties:
Description: Dummy function, simply logs received S3 events
Handler: index.handler
Runtime: python2.7
Role: !GetAtt 'TestFunctionRole.Arn'
Timeout: 240
Code:
ZipFile: |
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def handler(event, context):
logger.info('Received event: %s' % json.dumps(event))
TestS3BucketEventPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
SourceAccount: !Ref 'AWS::AccountId'
FunctionName: !Ref 'TestEventFunction'
SourceArn: !GetAtt
- TestS3Bucket
- Arn
Principal: s3.amazonaws.com
TestS3Bucket:
Type: AWS::S3::Bucket
DependsOn: TestS3BucketEventPermission
Properties:
NotificationConfiguration:
LambdaConfigurations:
-
Function:
"Fn::GetAtt":
- 'TestEventFunction'
- 'Arn'
Event: s3:ObjectCreated:*When you attempt to launch this template, you receive the following error:
Circular dependency between resources: [TestS3BucketEventPermission, TestS3Bucket].
This is because TestS3BucketEventPermission needs to know the ARN of the TestS3Bucket before it can create TestEventFunction. However, the TestS3Bucket’s NotificationConfiguration needs to know the ARN of TestEventFunction before it can create the TestS3Bucket, thus creating a circular dependency.
If you change the DependsOn property in the TestS3Bucket to point to the TestEventFunction instead, you run into a different error: “Unable to validate the following destination configurations.”
Solution using custom resource
We can avoid the circular dependency demonstrated earlier by creating the S3 bucket without any notification configuration. Then we set up the notification after the Lambda function, Lambda permission, and S3 bucket resources have been created. This can be done manually using the CLI, but CloudFormation custom resources provide a good mechanism to include the logic as part of your automated deployment lifecycle in CloudFormation itself.
In the following example, I create a new Lambda function, ApplyBucketNotificationFunction, that takes an S3 bucket and a Lambda function ARN as event inputs and applies NotificationConfiguration to the S3 bucket with the provided Lambda function.
Note: I have included the code inline for demonstration. You could source the Lambda code zip file from S3 to enable better code maintainability.
AWSTemplateFormatVersion: '2010-09-09'
Description: Sample template that demonstrates setting up Lambda event notification
for an S3 bucket using a custom resource
Resources:
TestFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Path: /
TestEventFunction:
Type: AWS::Lambda::Function
Properties:
Description: Dummy function, simply logs received S3 events
Handler: index.handler
Runtime: python2.7
Role: !GetAtt 'TestFunctionRole.Arn'
Timeout: 240
Code:
ZipFile: |
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def handler(event, context):
logger.info('Received event: %s' % json.dumps(event))
TestS3BucketEventPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
SourceAccount: !Ref 'AWS::AccountId'
FunctionName: !Ref 'TestEventFunction'
SourceArn: !GetAtt
- TestS3Bucket
- Arn
Principal: s3.amazonaws.com
TestS3Bucket:
Type: AWS::S3::Bucket
DependsOn: TestEventFunction
ApplyNotificationFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Path: /
Policies:
- PolicyName: S3BucketNotificationPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowBucketNotification
Effect: Allow
Action: s3:PutBucketNotification
Resource:
- !Sub 'arn:aws:s3:::${TestS3Bucket}'
- !Sub 'arn:aws:s3:::${TestS3Bucket}/*'
ApplyBucketNotificationFunction:
Type: AWS::Lambda::Function
Properties:
Description: Dummy function, just logs the received event
Handler: index.handler
Runtime: python2.7
Role: !GetAtt 'ApplyNotificationFunctionRole.Arn'
Timeout: 240
Code:
ZipFile: |
import boto3
import logging
import json
import cfnresponse
s3Client = boto3.client('s3')
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def addBucketNotification(bucketName, notificationId, functionArn):
notificationResponse = s3Client.put_bucket_notification_configuration(
Bucket=bucketName,
NotificationConfiguration={
'LambdaFunctionConfigurations': [
{
'Id': notificationId,
'LambdaFunctionArn': functionArn,
'Events': [
's3:ObjectCreated:*'
]
},
]
}
)
return notificationResponse
def create(properties, physical_id):
bucketName = properties['S3Bucket']
notificationId = properties['NotificationId']
functionArn = properties['FunctionARN']
response = addBucketNotification(bucketName, notificationId, functionArn)
logger.info('AddBucketNotification response: %s' % json.dumps(response))
return cfnresponse.SUCCESS, physical_id
def update(properties, physical_id):
return cfnresponse.SUCCESS, None
def delete(properties, physical_id):
return cfnresponse.SUCCESS, None
def handler(event, context):
logger.info('Received event: %s' % json.dumps(event))
status = cfnresponse.FAILED
new_physical_id = None
try:
properties = event.get('ResourceProperties')
physical_id = event.get('PhysicalResourceId')
status, new_physical_id = {
'Create': create,
'Update': update,
'Delete': delete
}.get(event['RequestType'], lambda x, y: (cfnresponse.FAILED, None))(properties, physical_id)
except Exception as e:
logger.error('Exception: %s' % e)
status = cfnresponse.FAILED
finally:
cfnresponse.send(event, context, status, {}, new_physical_id)
ApplyNotification:
Type: Custom::ApplyNotification
Properties:
ServiceToken: !GetAtt 'ApplyBucketNotificationFunction.Arn'
S3Bucket: !Ref 'TestS3Bucket'
FunctionARN: !GetAtt 'TestEventFunction.Arn'
NotificationId: S3ObjectCreatedEventConclusion
At the beginning of this blog post we saw that the Lambda permission (TestS3BucketEventPermission) and the NotificationConfiguration of the S3 bucket (TestS3Bucket) resources were dependent on each other, thus CloudFormation couldn’t create either resource.
I provided a solution example that avoids this circular dependency. I created the TestS3Bucket first without notification configurations. After that, I used a custom resource ApplyNotification that invokes the ApplyBucketNotificationFunction, which adds the Lambda notification configuration to the S3 bucket after both TestS3Bucket and TestEventFunction have been created.
There are two advantages to this solution:
First, S3 bucket names have to globally unique, so any solution that requires hardcoding bucket names runs the risk of indeterminate deployment errors in the event of a name collision. This approach prevents that scenario because it does not require S3 buckets to be given a fixed name.
Second, by using the custom resource I have made applying the bucket notification a part of the CloudFormation deployment. This eliminates any need for manual or scripting work necessary to set up the bucket notification after the stack has successfully deployed.
References
You can access code for the two templates below.
About the author
Vinod Shukla is a Partner Solutions Architect at Amazon Web Services. As part of the AWS Quick Starts team, he enjoys working with partners providing technical guidance and assistance in building gold-standard reference deployments.