Why is my CloudFormation stack stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state?

Lesedauer: 5 Minute
1

My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state. I want to get my stack into the UPDATE_ROLLBACK_COMPLETE or UPDATE_COMPLETE state.

Short description

A CloudFormation stack gets stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state when:

  • CloudFormation is still in the process of removing old resources, or can't remove those resources due to a dependency issue.
  • Resource cleanup is paused because a sibling stack of a nested stack fails to update or roll back.

Resolution

Check the status of your stack

  1. Open the CloudFormation console.
  2. In the navigation pane, choose Stacks.
  3. Choose the stack that's stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state.
  4. Check if your stack has a NESTED label next to its name.
  5. Choose the Resources tab for your stack, and then check if there are any resources in the DELETE_IN_PROGRESS state.
    Note: The Status column shows the resource state.

If your stack has a NESTED label next to its name, then complete the steps in the Troubleshoot nested stacks section.

If your stack doesn't have a label next to its name, then complete the steps in the Troubleshoot non-nested stacks section.

Troubleshoot nested stacks

If your stack has one or more resources in the DELETE_IN_PROGRESS state, then check the status of the resource. Or, follow the steps in the Check the AWS CloudTrail logs section of Why is my CloudFormation stack stuck in the state CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, or DELETE_IN_PROGRESS?

Your resource could be stuck in the DELETE_IN_PROGRESS state for the following reasons:

  • The resource requires additional time to be deleted. For example, resources like AWS::RDS::DBInstance and AWS::CloudFront::Distribution take longer to get deleted than other resources.
  • Other resources are dependent on your resource. For example, an Amazon Elastic Compute Cloud (Amazon EC2) security group could be used by other resources outside of your CloudFormation stack. Use the AWS Management Console to check that there are no dependencies manually created for the resource.

Note: CloudFormation attempts to delete a resource three times before skipping the resource and continuing the cleanup process.

If your stack has no resources in the DELETE_IN_PROGRESS state, then complete the following steps:

  1. Open the CloudFormation console.
  2. In the navigation pane, choose Stacks, and then choose your stack.
  3. In the Overview tab for your stack, choose the stack in the Root stack section.
  4. Check the status of the root stack in the Status section.

If the root stack is in the UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS state, then other resources dependent on your stack are still being updated. After all the dependent resources are updated (including resources in nested stacks), then CloudFormation starts the cleanup process.

If the root stack is in the UPDATE_ROLLBACK_FAILED state, identify the first resource that failed to roll back after the stack entered the UPDATE_ROLLBACK_IN_PROGRESS state. To identify this resource, check the Events tab of the root stack in the CloudFormation console.

If the resource that failed to roll back isn't a nested stack, then complete the steps in How can I get my CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

If the resource that failed to roll back is a nested stack, then complete the following steps:

  1. Identify the first resource that failed to roll back after the nested stack entered the UPDATE_ROLLBACK_IN_PROGRESS state. To identify this resource, check the Events tab of the nested stack in the CloudFormation console. If the resource is still a nested stack, then repeat this step for the nested stack until you identify the resource that failed to roll back that's not a nested stack.
  2. Continue the rollback process to bring the stack to a stable state.

Important: If you skip a resource during a ContinueUpdateRollback operation, CloudFormation sets the status of specified resources to UPDATE_COMPLETE. Then, CloudFormation continues to roll back the stack. After the rollback is complete, the state of the skipped resources is inconsistent with the state of the resources in the stack template. Before updating the stack again, update the resources to be consistent with each other. Otherwise, subsequent stack updates can fail, and the stack becomes unrecoverable.

Troubleshoot non-nested stacks

If your stack has one or more resources in the DELETE_IN_PROGRESS state, then check the status of the resource by using the AWS Management Console. Or, follow the steps in the Check the AWS CloudTrail logs section of Why is my CloudFormation stack stuck in the state CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, or DELETE_IN_PROGRESS?

Note: The Status column shows the resource state.

If there are no resources in the DELETE_IN_PROGRESS state, then look for operational issues using the AWS Service Health Dashboard for CloudFormation.


Related information

Working with nested stacks

Update rollback failed

AWS OFFICIAL
AWS OFFICIALAktualisiert vor 2 Jahren
4 Kommentare

So you get down to the bottom of the troubleshooter, "If there are no resources in the DELETE_IN_PROGRESS state, then look for operational issues using the AWS Service Health Dashboard for CloudFormation. " - there's nothing reported there, but my problem is that the stack is still in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS! Pretty useless troubleshooter!

Mike
beantwortet vor 6 Monaten

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
beantwortet vor 6 Monaten

Hi, I have a Cloudformation stack stuck in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state, however, all the resources are in the CREATE_COMPLETE or UPDATE_COMPLETE status. Thanks

Pedro
beantwortet vor 4 Monaten

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
beantwortet vor 4 Monaten