Integration & Automation
Handling circular dependency errors in AWS CloudFormation
Getting an error message is always a frustrating experience, and it can be more frustrating as you are learning to use a service or piece of technology that is new to you. A common error that you may get when working with AWS CloudFormation is the circular dependency error:
In this blog post I will review why this error occurs, a couple of scenarios where this error pops up, and how you can resolve these errors.
What is a circular dependency?
A circular dependency, as the name implies, means that two resources are dependent on each other or that a resource is dependent on itself:
Resource A is dependent on Resource B, and Resource B is dependent on Resource A. When AWS CloudFormation assesses that this type of condition exists, you will get a circular dependency error because AWS CloudFormation is unable to clearly determine which resource should be created first.
To understand why this can happen, let’s review some basics of AWS CloudFormation. When you define multiple resources in a template, AWS CloudFormation tries to create those resources in a parallel fashion (for speed) while also trying to extract correct dependencies based on the use of references when necessary. However, you can control resource-creation order by utilizing the DependsOn attribute within a resource. Using DependsOn, you can specify that Resource A needs to be created before Resource B. AWS CloudFormation also has implicit dependencies when utilizing the intrinsic function Ref. You can use Ref to reference information from another resource. When Resource A has a Ref in its properties to Resource B, Resource B is created before Resource A.
When you get a circular dependency error, the first step is to examine the resources that are outlined and ensure that AWS CloudFormation can determine what resource order it should take. In some cases, this may mean creating a resource and then modifying it after it has been created or supplying information to the resource in another fashion. Let’s review two examples and how to remedy them. You can find these examples in our GitHub repo.
Simple circular dependency example
In this first example, it is a bit easier to detect the circular dependency. There are cases where you may want to create a security group within AWS CloudFormation and assign an Amazon Elastic Compute Cloud (Amazon EC2) instance to it. In the security group, you may want a rule where all the members in the security group trust one another on a specific port.
The SimpleNonWorking.yaml template in the repo creates an Amazon EC2 instance and then places the EC2 instance in a security group that trusts all members of the security group on port 80. However, this template will generate a circular dependency error.
The next image illustrates where the circular dependency resides in the template.
You can see that security group rules have a Ref to the security group itself. This means this resource needs to be created before it can add the rule defined. Because the security group and rules are being defined in the same resource, we have created a circular dependency—in this case, a resource that is dependent on itself.
To solve this problem, we will want to separate the security group rule creation out to its own resource, as opposed to combining it in the security group resource creation. We want to create the security group first, then create a security group ingress rule that Refs the already-created security group to assign the rule. This will allow AWS CloudFormation to create the security group and assign it to the EC2 instance, and then assign the rule, therefore avoiding a circular dependency error. The next figure demonstrates the modified SimpleWorking.yaml template. The order in this example mitigates the circular dependency. First the security group is created, and then in parallel it is assigned to the instance and the ingress rule is added to the security group.
In this example, we hit a circular dependency because we used Ref, and we used Ref because in this case we would not know the security group ID beforehand, as it was being automatically generated. Hardcoding a value into an AWS CloudFormation template is not considered a best practice for reusability. To avoid hardcoding names, we can allow AWS CloudFormation to autogenerate names for us, but this can sometimes lead to circular dependency errors as well. In our next example, instead of moving the resource out, we will assign a name by using pseudo parameters to avoid the circular dependency.
Complex circular dependency example
Triggering AWS Lambda from an Amazon Simple Storage Service (Amazon S3) PUT Object is a common pattern. The template called LessSimpleNonWorking.yaml attempts to create the following:
- a Lambda function
- an S3 bucket
- a bucket notification that triggers the Lambda function on PUT Object API calls
However, it will generate a circular dependency error. The following diagram illustrates the circular dependency as well as the code that generates it.
In this example, it’s a little harder to detect the circular dependency because it is chained through several resources. The AWS CloudFormation template is using the AWS::Serverless transform to create a bucket that includes a bucket NotificationConfiguration property in the resource. The bucket notification is dependent on the Lambda function and the Lambda function is dependent on the execution role, which is dependent on the S3 bucket. Additionally, the sample used in this case uses a transform, so it’s harder to determine where the issue lies.
These types of dependencies can be the most frustrating to track down. However, trying to visualize what is happening as above will help to locate the circular dependency. In this case, we can break the dependency by setting the bucket name on creation, instead of allowing AWS CloudFormation to autogenerate the S3 bucket name. Then we also provide that name to the role, removing the bucket resource reference and thus the dependency loop. In the workaround, we have used pseudo parameters to specify a bucket name instead of autogenerating it, so we are still complying with the best practice of avoiding hardcoding bucket names in our template. Since we know the bucket name, in the policy section we can provide it the proper Amazon Resource Name (ARN).
For more information on resolving circular dependencies when deploying Lambda functions, you can check out the Knowledge Center, and you can also look at this blog post for another way to handle circular dependency errors.
Conclusion
In this blog post we reviewed what a circular dependency is, and what can cause it. We also examined two examples of circular dependency and how to resolve them. I hope this information will help you figure out how to resolve a circular dependency the next time you run across one.
We have established how in the Infrastructure & Automation blog we will cover real world examples and experience that we have run across. In sharing our experience we hope it can assist you in automating yours.