How do I troubleshoot Elastic Load Balancing "Connection timed out" errors?

Last updated: 2022-04-08

I'm using Elastic Load Balancing (ELB) and receiving a "Connection timed out" error. How do I troubleshoot a connection timeout between my load balancer and a client?

Resolution

Verify that your load balancer scheme is internet-facing

If the client is trying to connect over the internet, verify that the ELB scheme is internet-facing.

Confirm that a listener is configured for the load balancer port

Verify the configuration of the listeners for your load balancer port that the client connects to:

Confirm that the security groups for your target instances allow traffic

Application Load Balancers and Classic Load Balancers

Check the settings of the security group attached to the load balancer. Confirm that traffic is allowed in both directions for the listener and health check ports. For more information, see Security groups for your Application Load Balancer and Configure security groups for your Classic Load Balancer.

Network Load Balancers

If you register EC2 instances as targets, confirm that the security groups for these instances allow traffic on both the listener and health check ports. If you register targets by IP address, then be sure that the security group for the targets allows traffic from the NLB nodes. For more information, see Register targets with your target group (Network Load Balancer).

If you're using Client IP preservation, confirm that the target security group allows traffic from the client IP addresses. For more information, see Client IP preservation.

Verify that the network ACL allow the required traffic

Verify that the network access control list (network ACL) associated with the subnets where load balancer nodes are present:

  • Allows access on the port being accessed
  • Allows access to particular client IP addresses in the inbound rules.

Network ACLs are stateless, so you must allow access in both inbound and outbound rules. Also confirm that all the ephemeral ports are allowed in your outbound rules. Allowing only some ephemeral ports can lead to intermittent connection issues. For more information, see Network ACLs for load balancers in a VPC.

Confirm that route tables are configured correctly

For internet-facing load balancers, confirm that the route tables associated with the subnets where load balancer nodes are present has an internet gateway.

For internal load balancers using a private connection to the load balancer, there might be multiple route tables associated with different subnets. Connectivity issues can occur on individual nodes, causing intermittent issues. Check the route tables associated with all nodes. If connectivity fails for a node, then verify that the route table associated with the load balancers node subset is configured. Add routes as required.

Use the following command to test node connectivity:

nc –v <ELB IP Address> <Port>
telnet <ELB IP Address> <Port>

Note: Enter your information for ELB IP Address and Port.

Verify that the client isn't connecting to a terminated node (Application Load Balancer and Classic Load Balancer)

Confirm that the client isn't connecting to an Application Load Balancer or Classic Load Balancer node that is terminated. Verify that the client's IP addresses is included on the list of current IP addresses published in the ELB Domain Name System (DNS). Run the following dig command using your domain name for example.com:

$ dig +short example.com
35.154.42.229
13.126.64.179

Confirm that a client-side firewall isn't blocking traffic

If other troubleshooting methods don't resolve your connectivity issues, then a client-side firewall might be blocking traffic. To troubleshoot the issue, follow these steps:

  • If you're using VPC Flow Logs with metadata, check the logs of the load balancer node interfaces. Confirm that the SYN from the client IP address is received by the load balancer. Then, confirm that the SYN, ACK is sent back by the load balancer node to the client IP address. If SYN, ACK isn't returned from the load balancer node, then there might be an issue with that node. For an example of how to read flow logs, see TCP Flag sequence.
  • Take a packet capture on the machine that's running the client. Use the tcpdump utility to get network traffic for analysis. If the connection fails, update your firewall rules to allow the required traffic.