Why are properly functioning Amazon ECS tasks registered to ELB marked as unhealthy and replaced?

Last updated: 2019-06-20

Elastic Load Balancing is repeatedly flagging properly functioning Amazon Elastic Container Service (Amazon ECS) tasks as unhealthy. These incorrectly flagged tasks are stopped and new tasks are started to replace them. How can I troubleshoot this?

Resolution

Some Amazon ECS tasks have several dependencies and lengthy bootstrapping processes that can exceed the ELB health check grace period, even when functioning as intended. When Amazon ECS tasks don’t respond to ELB health checks within the grace period, they're flagged as unhealthy. To increase the health check grace period for your service:

  1. Open the AWS Management Console.
  2. In the navigation bar, choose Services, and then select ECS from the list.
  3. Select your service from the Service Name list.
  4. Choose Update.
  5. Choose Next step.
  6. On the Step 2: Configure network page, change the Health check grace period to an appropriate time period for your service (maximum is 7,200 seconds).
    Caution: To prevent delayed replacement of legitimately unhealthy Amazon ECS tasks, carefully estimate the grace period required for your lengthiest known tasks. When setting your grace period, consider all relevant factors such as bootstrap time and time to pull container images.
  7. Choose Next step, and then choose Update Service.

You can also increase the grace period:

  • Using the HealthCheckGracePeriodSeconds parameter defined in the AWS::ECS::Service resource in AWS CloudFormation.
  • By running the UpdateService command in the AWS Command Line Interface (AWS CLI) and increasing the --health-check-grace-period-seconds value.

Did this article help you?

Anything we could improve?


Need more help?