How do I troubleshoot 504 errors returned while using a Classic Load Balancer?
Last updated: 2018-03-21
I see HTTP 504 errors in Classic Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through a Classic Load Balancer. How do I fix this?
An HTTP 504 error is a HTTP status code that indicates a gateway or proxy has timed out. When troubleshooting, investigate the following:
Check your load balancer’s idle timeout, and then modify if necessary
The most common reason for a load balancer to return HTTP 504 errors is that a corresponding backend instance did not respond to the request within the currently configured idle timeout. By default, the idle timeout for Classic Load Balancer is 60 seconds.
If CloudWatch metrics are enabled, check CloudWatch metrics for your load balancer. If latency data points are equal to your currently configured load balancer timeout value, and there are data points in the HTTPCode_ELB_5XX metric, then at least one request has timed out.
To resolve this, either modify the idle timeout for your load balancer so that the HTTP request will be completed within the idle timeout period, or tune your application to respond more quickly.
Make sure that your backend instances keep connections open
If a backend instance closes a TCP connection to the load balancer before the load balancer has reached its idle timeout value, the load balancer might not be able to fulfill the request, generating an HTTP 504 error.
To resolve this, enable keep-alive settings on your backend instances, and set the keep-alive timeout to a value greater than the load balancer’s idle timeout.
(Apache only) Disable TCP_DEFER_ACCEPT
When TCP_DEFER_ACCEPT is enabled for Apache backend instances, the load balancer initiates a connection by sending a SYN to the backend instance. The backend instance then responds with a SYN-ACK, and the load balancer sends an empty ACK to the backend instance.
Because the last ACK is empty, the back end does not accept the ACK, and instead resends a SYN-ACK to the load balancer. This can result in a subsequent SYN retry timeout. When the backend instance closes the connection without sending a FIN or RST to the load balancer, the load balancer considers the connection to be established even though it is not. Then when the load balancer sends requests through this TCP connection, the back end responds with an RST, generating a 504 error.
To resolve this, set both AcceptFilter http and AcceptFilter https to none.
(Apache only) Disable the event MPM, and optimally configure the prefork and worker MPMs
The event MPM should not be used on backend instances that are registered to a load balancer, because the Apache backend dynamically closes connections, which results in HTTP 504 errors being sent to the clients.
|mod_prefork MPM||mod_worker MPM|