How can I get my Amazon ECS tasks running using the Amazon EC2 launch type to pass the Application Load Balancer health check in Amazon ECS?

Last updated: 2020-11-12

An Application Load Balancer health check for an Amazon Elastic Compute Cloud (Amazon EC2) instance in Amazon Elastic Container Service (Amazon ECS) is returning an unhealthy status. I want my EC2 instance to pass the health check.

Short description

To pass the Application Load Balancer health check, confirm the following:

  • The application in your ECS container returns the correct response code.
  • The security groups attached to your load balancer and container instance are correctly configured.
  • The advanced health check settings of your target group are correctly configured.

Note: An ECS task can return an unhealthy status for many reasons. If the following steps don't resolve your issue, see Troubleshooting service load balancers.

Tip: To find out what stopped your ECS task, see Checking stopped tasks for errors.

Resolution

Confirm that the application in your ECS container returns the correct response code

When the load balancer sends an HTTP GET request to the health check path, the application in your ECS container should return the default 200 OK response code.

Note: If you use an Application Load Balancer, you can update the Matcher setting to a response code other than 200. For more information, see Health checks for your target groups.

1.    Connect to your container instance using SSH.

2.    (Optional) Install curl with the command appropriate for your system.

For Amazon Linux and other RPM-based distributions, run the following command:

sudo yum –y install curl

For Debian-based systems (such as Ubuntu), run the following command:

sudo apt-get install curl

3.    To get the container ID, run the following command:

docker ps

Note: The port for the local listener appears in the command output under PORTS at the end of the sequence after the arrow bracket.

4.    To get the IP address of the container, use the docker inspect command:

$ IPADDR=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' aabbccddeeff)

Note: The IP address of the container is saved to IPADDR. This will return an IP if using the BRIDGE network mode. If you use the AWSVPC or HOST network mode, then use the task IP assigned to the task ENI or host that the task is exposed through.

5.    To get the status code, run a curl command that includes IPADDR and the port of the local listener.

See the following example of a container listening on port 8080 with the health check path of /health:

curl -I http://${IPADDR}:8080/health

The command should return 200 OK.

If you receive a non-HTTP error message, then your application isn't listening to HTTP traffic. If you receive an HTTP status code different from what you specified in the Matcher setting, then your application is listening but not returning a status code for a healthy target.

Correctly configure the security groups attached to your load balancer and container instance

As a best practice, configure one security group for your load balancer and another security group for your container instance. By following this best practice, you allow all traffic between your load balancers and container instances. Also, enable your container instances to accept traffic on the host port that is specified for the task.

1.    Confirm that the security group associated with your load balancer allows all egress traffic to the security group associated with your container instance.

2.    Confirm that the security group associated with your container instance allows all ingress traffic on the task host port range from the security group associated with your load balancer.

Important: If you're using dynamic port mapping, the service is exposed on the dynamic port (typically ports 32768-65535) rather than on the host port. For this reason, confirm that your container instance security group reflects the ephemeral port range in the ingress rules for the load balancer as a source.

To check the security group associated with your load balancer, see Security groups for your Application Load Balancer.

Configure the advanced health check settings of your target group

To configure your advanced health check settings correctly, see Health checks for your target groups. When you configure your advanced health check settings, pay close attention to the following steps:

1.    Open the Amazon EC2 console, choose Target Groups, and then choose your target group.

Important: Use a new target group. Avoid adding targets to the target group manually, because Amazon ECS automatically registers and de-registers containers with the target group.

2.    Choose the Health checks view.

3.    For Port, choose traffic port.

Note: If you choose Override, then confirm that the port specified matches the task host port.

Confirm that your load balancer is configured in the same Availability Zones as your container instances

To get the Availability Zones that your load balancer is configured for:

1.    Open the Amazon EC2 console.

2.    From the navigation pane, in the Load Balancing section, choose Load Balancers.

3.    Select the load balancer that you're using with your Amazon ECS service.

4.    On the Description tab, in the Availability Zones field, note the listed Availability Zones. 

To get the Availability Zones that your container instances are configured for:

1.    Open the Amazon EC2 console.

2.    In the navigation pane, in the Auto Scaling section, choose Auto Scaling Groups

3.    Select the container instance Auto Scaling group that is associated to your cluster.

4.    On the Details tab, in the Network section, confirm that the Availability Zones listed match the Availability Zones listed for your load balancer.


Did this article help?


Do you need billing or technical support?