How do I troubleshoot the container health check failures for Amazon ECS tasks?
Last updated: 2022-03-11
My Amazon Elastic Container Service (Amazon ECS) task is failing the container heath check. How do I resolve this?
If you receive the following error, then the Amazon ECS containers in your task are using health checks that your service can't pass:
(service AWS-Service) (task ff3e71a4-d7e5-428b-9232-2345657889) failed container health checks
Note: If you're experiencing Elastic Load Balancing (ELB) health check issues, see How can I get my Amazon ECS tasks running to pass the Application Load Balancer health check in Amazon ECS?
To troubleshoot Amazon ECS container health check failures, try the following tips:
- Test the container locally to make sure that it passes the container health checks before provisioning to Amazon ECS.
- Confirm that the command that you're passing to the container is correct and that you're using the correct syntax for your Amazon ECS tasks.
- Make sure that your container has enough time to instantiate.
- If your Amazon ECS task has been running for a while, check your application logs and Amazon CloudWatch logs.
Test the container locally to make sure it passes the container health check
Before provisioning your container to Amazon ECS, make sure that your container can work as expected and pass the specified container health check. You can test your container with the Dockerfile HEALTHCHECK configuration on the Docker website. After your container passes the health check defined in Dockerfile, you can specify the health check configuration in the task definition. Specifying the health check configuration allows the Amazon ECS container agent to monitor and report the health check specified.
Note: Amazon ECS doesn't monitor Docker health checks that are embedded in a container image and aren't specified in the container definition. Health check parameters that are specified in a container definition override any Docker health checks that exist in the container image.
Confirm that you're using the correct syntax for your Amazon ECS tasks
Make sure that you're using the correct commands and syntax for your Amazon ECS tasks.
For example, if you use the AWS Management Console JSON panel, the AWS Command Line Interface (AWS CLI), or APIs, then enclose the list of commands in brackets, like this:
["CMD-SHELL", "curl -f http://localhost/ || exit 1"]
If you're using the AWS Management Console to edit your ECS task, then you don't need to include the brackets:
"CMD-SHELL", "curl -f http://localhost/ || exit 1"
Also, make sure that you're not separating the health check command with double quotes, such as ["CMD-SHELL", "healthcheck.sh", "||", "exit 1"]. Instead, use the following command syntax:
["CMD-SHELL", "healthcheck.sh || exit 1"]
Make sure that your container has enough time to instantiate
If your container takes a while to initiate, then your container could fail the container health check. Try setting the startPeriod in the advanced container definition parameter. This gives your Amazon ECS container enough time to bootstrap before any failed health checks are included in the maximum number of retries.
Check your application logs and Amazon CloudWatch logs if the task has been running for a while
If your Amazon ECS container has been running for a while and it fails the container health check, check your application logs. If your Amazon ECS task uses awslogs log driver, then check your application logs on Amazon CloudWatch.
Note: AWS Fargate is a managed service. Therefore, you can't access the underlying infrastructure. To troubleshoot, launch your Amazon ECS tasks in Amazon Elastic Compute Cloud (Amazon EC2). Then, connect to your Amazon EC2 instances using SSH. You can also try using Amazon ECS Exec to interact directly with your ECS containers.