How can I get my Amazon EC2 instance to pass the Application Load Balancer health check in Amazon ECS?

Last updated: 2020-04-22

An Application Load Balancer health check for an Amazon Elastic Compute Cloud (Amazon EC2) instance in Amazon Elastic Container Service (Amazon ECS) is returning an unhealthy status. How can I get my EC2 instance to pass the health check?

Short Description

To pass the Application Load Balancer health check, confirm the following:

  • The application in your ECS container returns the correct response code.
  • The security groups attached to your load balancer and container instance are correctly configured.
  • The advanced health check settings of your target group are correctly configured.

Note: An ECS task can return an unhealthy status for many reasons. If the following steps don't resolve your issue, see Troubleshooting service load balancers.

Tip: To find out what stopped your ECS task, see Checking stopped tasks for errors.

Resolution

Confirm that the application in your ECS container returns the correct response code

When the load balancer sends an HTTP GET request to the health check path, the application in your ECS container should return the default 200 OK response code.

Note: If you use an Application Load Balancer, you can update the Matcher setting to a response code other than 200. For more information, see Health Checks for Your Target Groups.

1.    Connect to your container instance using SSH.

2.    (Optional) Install curl with the command appropriate for your system.

For Amazon Linux and other RPM-based distributions, run the following command:

sudo yum –y install curl

For Debian-based systems (such as Ubuntu), run the following command:

sudo apt-get install curl

3.    To get the container ID, run the following command:

docker ps

Note: The port for the local listener appears in the command output under PORTS, at the end of the sequence after the arrow bracket.

4.    To get the IP address of the container, use the docker inspect command. See the following example:

$ IPADDR=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' aabbccddeeff)

Note: The IP address of the container is saved to IPADDR.

5.    To get the status code, run a curl command that includes IPADDR and the port of the local listener.

See the following example of a container listening on port 8080 with the health check path of /health:

curl -v http://${IPADDR}:8080/health

The command should return 200 OK.

If you receive a non-HTTP error message, then your application isn't listening to HTTP traffic. If you receive an HTTP status code different from what you specified in the Matcher setting, then your application is listening but not returning a status code for a healthy target.

Correctly configure the security groups attached to your load balancer and container instance

As a best practice, configure one security group for your load balancer and another security group for your container instance. By following this best practice, you allow all traffic between your load balancers and container instances. Also, enable your container instances to accept traffic on the ephemeral port range that's used for dynamic host port mapping.

1.    Confirm that the security group associated with your load balancer allows all egress traffic to the security group associated with your container instance.

2.    Confirm that the security group associated with your container instance allows all ingress traffic on the ephemeral port range (typically ports 32768-65535) from the security group associated with your load balancer.

Important: If you declare the host port in your task definition, the service is exposed on the specified port rather than in the ephemeral port range. For this reason, confirm that your security group reflects the specified host port instead of the ephemeral port range.

To check the security group associated with your load balancer, see Security Groups for Your Application Load Balancer.

Correctly configure the advanced health check settings of your target group

To configure your advanced health check settings correctly, see Health Checks for Your Target Groups. When you configure your advanced health check settings, pay close attention to the following steps:

1.    Open the Amazon EC2 console, choose Target Groups, and then choose your target group.

Important: Be sure to use a new target group. Avoid adding targets to the target group manually, because Amazon ECS automatically registers and de-registers containers with the target group.

2.    Choose the Health checks view.

3.    For Port, choose traffic port.

Note: If you choose Override, then health check traffic won't be routed correctly.


Did this article help you?

Anything we could improve?


Need more help?