Why is my AWS CloudFormation stack stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state?

Last updated: 2020-11-06

I want to know why my AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state. I want to get my stack into the UPDATE_ROLLBACK_COMPLETE or UPDATE_COMPLETE state.

Short description

An AWS CloudFormation stack gets stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state when:

  • AWS CloudFormation is still in the process of removing old resources, or can't remove those resources due to a dependency issue.
  • Resource cleanup is put on pause because a sibling stack of a nested stack fails to update or roll back.

Resolution

Check the status of your stack

  1. Open the AWS CloudFormation console.
  2. In the navigation pane, choose Stacks.
  3. Choose the stack that's stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state.
  4. Check if your stack has a NESTED label next to its name.
  5. Choose the Resources tab for your stack, and then check if there are any resources in the DELETE_IN_PROGRESS state.
    Note:
    The Status column shows the resource state.

If your stack has a NESTED label next to its name, complete the steps in the Troubleshoot nested stacks section.

If your stack doesn't have a label next its name, then complete the steps in the Troubleshoot non-nested stacks section.

Troubleshoot nested stacks

If your stack has one or more resources in the DELETE_IN_PROGRESS state, then use the AWS Management Console to check the status of the resource. Or, follow the steps in the Check the AWS CloudTrail logs section of Why is my AWS CloudFormation stack stuck in the state CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, or DELETE_IN_PROGRESS?

Your resource could be stuck in the DELETE_IN_PROGRESS state for the following reasons:

  • The resource requires additional time to be deleted. For example, resources like AWS::RDS::DBInstance and AWS::CloudFront::Distribution take longer to get deleted than other resources.
  • Other resources are dependent on your resource. For example, an Amazon Elastic Compute Cloud (Amazon EC2) security group could be used by other resources outside of your AWS CloudFormation stack. Use the AWS Management Console to check that there are no dependencies manually created for the resource.

Note: AWS CloudFormation attempts to delete a resource three times before skipping the resource and continuing the cleanup process.

If your stack has no resources in the DELETE_IN_PROGRESS state, then complete the following steps:

  1. Open the AWS CloudFormation console.
  2. In the navigation pane, choose Stacks, and then choose your stack.
  3. In the Overview tab for your stack, choose the stack in the Root stack section.
  4. Check the status of the root stack in the Status section.

If the root stack is in the UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS state, then other resources dependent on your stack are still being updated. After all the dependent resources are updated (including resources in nested stacks), then AWS CloudFormation starts the cleanup process.

If the root stack is in the UPDATE_ROLLBACK_FAILED state, identify the first resource that failed to roll back after the stack entered the UPDATE_ROLLBACK_IN_PROGRESS state. To do identify this resource, check the Events tab of the root stack in the AWS CloudFormation console.

If the resource that failed to roll back is not a nested stack, then complete the steps in How can I get my AWS CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

If the resource that failed to roll back is a nested stack, then complete the following steps:

  1. Identify the first resource that failed to roll back after the nested stack entered the UPDATE_ROLLBACK_IN_PROGRESS state by checking the Events tab of the nested stack in the AWS CloudFormation console. If the resource is still a nested stack, then repeat this step for the nested stack until you identify the resource that failed to roll back that's not a nested stack.
  2. Continue the rollback process to bring the stack to a stable state.

Important: If you skip a resource during a ContinueUpdateRollback operation, AWS CloudFormation sets the status of specified resources to UPDATE_COMPLETE. Then, AWS CloudFormation continues to roll back the stack. After the roll back is complete, the state of the skipped resources is inconsistent with the state of the resources in the stack template. Before updating the stack again, update the resources to be consistent with each other. Otherwise, subsequent stack updates can fail, and the stack becomes unrecoverable.

Troubleshoot non-nested stacks

If your stack has one or more resources in the DELETE_IN_PROGRESS state, then check the status of the resource by using the AWS Management Console. Or, follow the steps in the Check the AWS CloudTrail logs section of Why is my AWS CloudFormation stack stuck in the state CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, or DELETE_IN_PROGRESS?

Note: The Status column shows the resource state.

If there are no resources in the DELETE_IN_PROGRESS state, then look for operational issues using the AWS Service Health Dashboard for AWS CloudFormation.


Did this article help?


Do you need billing or technical support?