How can I get my AWS::ECS::Service resources out of UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS status?

Last updated: 2019-07-16

My AWS CloudFormation stack update to the AWS::ECS::Service resource got stuck in UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS status. How can I stabilize the stack and get my service to launch new tasks?

Short Description

Your Amazon Elastic Container Service (Amazon ECS) service can get stuck in UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS status when the service fails to launch tasks.

Here are some common reasons why an Amazon ECS service can fail to launch new tasks:

  • Container image issues
  • A lack of necessary resources for launching tasks
  • A health check failure on a load balancer
  • Instance configuration or Amazon ECS container agent issues

An Amazon ECS service that fails to launch tasks causes AWS CloudFormation to get stuck in UPDATE_IN_PROGRESS status. Then, AWS CloudFormation waits for several hours before rolling back to a previous configuration (previous task definition revision). If the issue that's causing stack failure continues during stack rollback to a previous configuration, then the stack gets stuck in UPDATE_ROLLBACK_IN_PROGRESS status. Finally, the stack changes to UPDATE_ROLLBACK_FAILED status.

It can take the AWS CloudFormation stack several hours to stabilize. To stabilize your stack more quickly, complete the following steps.

Important: The following resolution is intended to help you stabilize an AWS CloudFormation stack quickly without waiting for the stack to time out. The resolution isn't intended for production environments, as the Amazon ECS service is out of sync with the known state of AWS CloudFormation. To sync resources between your Amazon ECS service and the AWS CloudFormation stack, you must perform an error-free update on the stack.

Resolution

Change the desired task count of the Amazon ECS service

1.    Open the Amazon ECS console.

2.    Choose your cluster.

3.    Select the service, and then choose Update.

4.    Set Number of tasks to 0, and then save the configuration.

Identify why the Amazon ECS service can't launch new tasks

1.    Open the Amazon ECS console.

2.    Choose your cluster.

3.    Select the service, and then choose Events.

Note: The Events section displays the reason why your service didn't stabilize.

4.    Choose a solution based on the issue that you have identified:
Your task failed Elastic Load Balancing (ELB) health checks.
A container marked as essential for the task definition exited or died.
You can't place a task because your container instance didn't meet the necessary requirements.
You receive a "cannot pull container image" error.


Did this article help you?

Anything we could improve?


Need more help?