How can I stop my Amazon ECS service from failing to stabilize in AWS CloudFormation?

Last updated: 2020-09-09

My Amazon Elastic Container Service (Amazon ECS) service fails to stabilize in AWS CloudFormation. I get the following error: "Service arn:aws:ecs:us-east-accountID:service/ServiceName did not stabilize."

Short description

A service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To confirm that a service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message. While AWS CloudFormation checks the status of the service, the stack that contains the service remains in the CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS state and can't be updated.

If you can't fix the underlying issue with your Amazon ECS service tasks immediately and you don't want to wait for the DescribeService API calls to time out, then you can manually force the state of the Amazon ECS service resource in AWS CloudFormation into a CREATE_COMPLETE state. To do this, manually set the desired count of the service to zero in the Amazon ECS console to stop running tasks. AWS CloudFormation then considers the update as successful, because the number of tasks equals the desired count of zero.

Important: Manually forcing AWS CloudFormation into a CREATE_COMPLETE state isn't a best practice for production services, because all tasks are stopped and doing this can cause a production outage.

Resolution

Verify resource creation

1.    In your AWS CloudFormation template, create an AWS::ECS::Service resource. For example:

Resources:           
ECSServiceA:
 Type: AWS::ECS::Service
Properties:
  DesiredCount: 1
  Cluster: awsExampleECSCluster
  LaunchType: EC2
  ServiceName: "MyNginxService2"
  TaskDefinition: NginxTask:1

2.    Open the AWS CloudFormation console, and then select your stack.

3.    Choose the Events tab, and then verify that your resource is being created.

Update the desired count of the service

You can update the desired count of the service to your original value with either the AWS Command Line Interface (AWS CLI) or Amazon ECS console.

Using the AWS CLI:

1.    To describe the service and list the service events, run the following command:

aws ecs describe-services --cluster awsExampleECSCluster --services MyNginxService2

2.    To update the desired count of the service, run the following command:

aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 0

3.    Update --desired-count to your original value.

Using the Amazon ECS console:

1.    Open the Amazon ECS console.

2.    In the navigation pane, choose Clusters, and then select the cluster that contains the ECS service that you created.

3.    Choose the Events tab, and then choose Update.

4.    For Number of tasks, enter 0.

The service now reaches a steady state and transitions the Amazon ECS service resource in AWS CloudFormation to CREATE_COMPLETE or UPDATE_COMPLETE.

5.    For Number of tasks, enter your original value.

Note: To make your AWS CloudFormation stack sync with the Amazon ECS service properties after you fix the issue with the underlying tasks, you must manually change desired count back to the original value as per the template.


Did this article help?


Do you need billing or technical support?