Why are properly functioning Amazon ECS tasks registered to ELB marked as unhealthy and replaced?

Last updated: 2022-09-19

Elastic Load Balancing (ELB) is repeatedly flagging properly functioning Amazon Elastic Container Service (Amazon ECS) tasks as unhealthy. These incorrectly flagged tasks are stopped, and then new tasks are started instead. How can I troubleshoot this?

Short description

Some Amazon ECS tasks have several dependencies and lengthy bootstrapping processes that can exceed the ELB health check grace period, even when functioning as intended. When Amazon ECS tasks don’t respond to ELB health checks within the grace period, they're flagged as unhealthy. To increase the health check grace period for your service, complete the steps below.

To troubleshoot ECS tasks failing an Application Load Balancer health check, see How can I get my Amazon ECS tasks running using the Amazon EC2 launch type to pass the Application Load Balancer health check in Amazon ECS?

Resolution

If no grace period is configured, then the service scheduler immediately replaces any targets marked as unhealthy. Change the grace period to allow more time for your Amazon ECS tasks to complete their processes and pass the health check.

  1. Open the AWS Management Console.
  2. In the navigation bar, choose Services, and then select ECS from the list.
  3. Select your service from the Service Name list.
  4. Choose Update.
  5. Choose Next step.
  6. On the Step 2: Configure network page, change the Health check grace period to an appropriate time period for your service (maximum is 2,147,483,647 seconds).
    Caution: To prevent delayed replacement of legitimately unhealthy Amazon ECS tasks, carefully estimate the grace period required for your lengthiest known tasks. When setting your grace period, consider all relevant factors such as bootstrap time and time to pull container images.
  7. Choose Next step, and then choose Update Service.

You can also increase the grace period:

  • Using the HealthCheckGracePeriodSeconds parameter defined in the AWS::ECS::Service resource in AWS CloudFormation.
  • By running the UpdateService command in the AWS Command Line Interface (AWS CLI) and increasing the --health-check-grace-period-seconds value.

Did this article help?


Do you need billing or technical support?