How do I troubleshoot 504 errors returned while using a Classic Load Balancer?

Last updated: 2022-10-07

I see HTTP 504 errors in Classic Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through a Classic Load Balancer. How do I fix this?

Resolution

An HTTP 504 error is an HTTP status code that indicates a gateway or proxy has timed out. When troubleshooting, investigate the following:

Check your load balancer’s idle timeout, and then modify if necessary

The most common reason for an HTTP 504 error is that a corresponding instance didn't respond to the request within the configured idle timeout. By default, the idle timeout for Classic Load Balancer is 60 seconds.

If CloudWatch metrics are turned on, check CloudWatch metrics for your load balancer. When latency data points are equal to your configured load balancer timeout value, and HTTPCode_ELB_5XX has data points, at least one request has timed out.

To resolve this, you can do one of two things:

  • Modify the idle timeout for your load balancer so that the HTTP request is completed within the idle timeout period.
  • Tune your application to respond more quickly.

Make sure that your backend instances keep connections open

When the backend instance closes a TCP connection to the load balancer before it reaches its idle timeout value, an HTTP 504 error might display.

To resolve this, activate keep-alive settings on your backend instances, and then set the keep-alive timeout to a value greater than the load balancer’s idle timeout.

(Apache only) Turn off TCP_DEFER_ACCEPT

When TCP_DEFER_ACCEPT is activated for Apache backend instances, the load balancer initiates a connection by sending a SYN to the backend instance. The backend instance then responds with a SYN-ACK, and the load balancer sends an empty ACK to the backend instance.

Because the last ACK is empty, the backend doesn't accept the ACK, and instead resends a SYN-ACK to the load balancer. This can result in a subsequent SYN retry timeout. When FIN or RST aren’t sent before the backend instance closes the connection, the load balancer considers the connection established, but it's not. Then, when the load balancer sends requests through this TCP connection, the backend responds with an RST, generating a 504 error.

To resolve this, set both AcceptFilter http and AcceptFilter https to none.

(Apache only) Turn off the event MPM, and optimally configure the prefork and worker MPMs

It’s a best practice to turn off event MPM on backend instances that are registered to a load balancer. Because the Apache backend dynamically closes connections, these closed connections might result in HTTP 504 errors.

For optimal performance when using the prefork and worker MPMs, and presuming the load balancer is configured with a 60-second idle timeout, use these values:

  mod_prefork MPM mod_worker MPM
Timeout 65 65
KeepAliveTimeout 65 65
KeepAlive On On
MaxKeepAliveRequests 10000 0
AcceptFilter http none none
AcceptFilter https none none