Clients experience high latency when connecting to web applications running on Amazon Elastic Compute Cloud (Amazon EC2) instances registered to a Classic Load Balancer. How do I troubleshoot this Elastic Load Balancing (ELB) latency?

High latency on a Classic Load Balancer can occur due to the following:

  • Network connectivity issues
  • Improper configuration of the Classic Load Balancer
  • High memory (RAM) utilization on the backend instances
  • High CPU utilization on the backend instances
  • Improper web server configuration on the backend instances
  • Problems with web application dependencies running on the backend instances, such as external databases or Amazon Simple Storage Service (Amazon S3) buckets

1.    Troubleshoot network connectivity issues for your Classic Load Balancer.  

2.    Be sure to configure your Classic Load Balancer correctly for your use case.

3.    Determine which backend instances are experiencing high latency by checking the access logs for your Classic Load Balancer. Look at backend_processing_time to find backend instances with latency issues.

To verify that a backend instance's web application server is experiencing high latency, use curl to measure the first byte response. For example:

[ec2-user@ip- ~]$ for X in `seq 6`; do curl -Ik -w "HTTPCode=%{http_code} TotalTime=%{time_total}\n" -so /dev/null; done

High Latency sample output:
HTTPCode=200 TotalTime=2.452
HTTPCode=200 TotalTime=1.035

Low latency sample output:
HTTPCode=200 TotalTime=0.515
HTTPCode=200 TotalTime=0.013

4.    Check the Average statistic of the CloudWatch Latency metric for your Classic Load Balancer. If the value is high, there is typically a problem with backend instances or application dependency servers instead of ELB. 

5.    Check the Maximum statistic of the Latency metric. If the value meets or exceeds the idle timeout value, requests are likely timing out. This issue results in an HTTP 504 error for clients.

6.    Check for patterns in the Latency metric. If the metric spikes at regular intervals or in a pattern, this might indicate performance problems with backend instances. This problem can happen due to overhead from scheduled tasks.

7.    Check the CloudWatch SurgeQueueLength metric of the ELB. If requests to the Classic Load Balancer exceed the maximum value (1024), requests are rejected and an HTTP 503 error is generated by the load balancer. The sum statistic of the SpilloverCount metric measures the total number of rejected requests. For more information, see How do I troubleshoot Elastic Load Balancing capacity issues?

8.    Check for possible memory issues by checking the Apache processes on your backend instances.

Example command:

watch -n 1 "echo -n 'Apache Processes: ' && ps -C apache2 --no-headers | wc -l && free -m"

Example output:

Every 1.0s: echo –n 'Apache Processes: ' && ps –C apache2 –no-
headers | wc -1 && free –m
Apache Processes: 27
          total     used     free     shared     buffers     cached
Mem:      8204      7445     758      0          385         4567
-/+ buffers/cache:  2402     5801
Swap:     16383     189      16194

9.    Check the CloudWatch CPUUtilization metric of your backend instances. Look for high CPU utilization or spikes in CPU utilization. For high CPU utilization, consider upgrading your instances to a larger instance type.

10.    Check the MaxClient setting for the web servers on your backend instances, which defines how many simultaneous requests the instance can serve. For instances with appropriate memory and CPU utilization experiencing high latency, consider increasing the MaxClient value.

Compare the number of processes generated by Apache (httpd) with the MaxClient setting. If the number of Apache processes frequently reaches the MaxClient value, consider increasing the value.

Example command:

[root@ip- conf]# ps aux | grep httpd | wc -l 15

Example output: 

<IfModule prefork.c>
StartServers         10
MinSpareServers      5
MaxSpareServers      10
ServerLimit          15
MaxClients           15
MaxRequestsPerChild  4000

11.    Check for dependencies of your backend instances that might be causing latency issues. These include, but aren’t limited to: shared databases, external resources (such as S3 buckets), external resource connections such as network address translation (NAT) instances, remote web services, and proxy servers.

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center.

Published: 2017-10-10

Updated: 2018-12-18