How can I stop my Amazon ECS service from failing to stabilize in AWS CloudFormation?

Last updated: 2019-05-29

My Amazon Elastic Container Service (Amazon ECS) service fails to stabilize in AWS CloudFormation. I get the following error: "Service arn:aws:ecs:us-east-accountID:service/ServiceName did not stabilize." How can I resolve this error?

Short Description

A service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To be sure that the service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message. While AWS CloudFormation checks the status of the service, the stack that contains the service remains in the CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS state and can't be updated.

If you can't fix the underlying issue with your ECS service tasks immediately and you don't want to wait for the DescribeService API calls to time out, then you can manually force the state of the ECS service resource in AWS CloudFormation into a CREATE_COMPLETE state. To do this, manually set the desired count of the service to zero in the Amazon ECS console to stop the running tasks. AWS CloudFormation then considers the update as successful, because the number of tasks equals the desired count of zero.

Important: Manually forcing AWS CloudFormation into a CREATE_COMPLETE state isn't recommended for production services, because all tasks are stopped and doing this can cause a production outage.

Resolution

Verify resource creation

1.    In your AWS CloudFormation template, create an AWS::ECS::Service resource. See the following example:

Resources:           
ECSServiceA:
 Type: AWS::ECS::Service
Properties:
  DesiredCount: 1
  Cluster: awsExampleECSCluster
  LaunchType: EC2
  ServiceName: "MyNginxService2"
  TaskDefinition: NginxTask:1

2.    Open the AWS CloudFormation console, and then select your stack.

3.    Choose the Events tab, and then verify that your resource is being created.

Update the desired count of the service

You can update the desired count of the service to your original value with either the AWS Command Line Interface (AWS CLI) or Amazon ECS console.

AWS CLI:

1.    To describe the service and list the service events, run the following command:

aws ecs describe-services --cluster awsExampleECSCluster --services MyNginxService2

2.    To update the desired count of the service, run the following command:

aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 0

3.    Update --desired-count to your original value.

Amazon ECS console:

1.    Open the Amazon ECS console.

2.    In the navigation pane, choose Clusters, and then select the cluster that contains the ECS service that you created.

3.    Choose the Events tab, and then choose Update.

4.    For Number of tasks, enter 0.

The service now reaches a steady state, and transitions the ECS service resource in AWS CloudFormation to CREATE_COMPLETE or UPDATE_COMPLETE.

5.    For Number of tasks, enter your original value.

Note: To make your AWS CloudFormation stack sync with the ECS service properties after you fix the issue with the underlying tasks, you must manually change desired count back to the original value as per the template.


Did this article help you?

Anything we could improve?


Need more help?