How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?

Last updated: 2022-12-14

I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?

Resolution

When your REST API requests return an HTTP 504 error status code, you must run various checks.

Check for IntegrationLatency spikes

Check if there's an IntegrationLatency spike in API Gateway by reviewing the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging.

For more information, see Set up Amazon CloudWatch API logging using the API Gateway console.

An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. Check the Lambda function's Duration performance metric to confirm this.

For more information, see Working with Lambda function metrics.

Review requests with CloudWatch Logs Insights

Use Amazon CloudWatch Logs Insights to review requests that resulted in 504 errors. To review the requests, on the CloudWatch console, in the navigation pane, choose Logs, Log Insights. Select your API Gateway log group. Then, set the relative time with one of the following queries:

parse @message '(*) *' as reqId, message
| filter message like /Method completed with status: \d\d\d/
| parse message 'Method completed with status: *' as status
| filter status = 504
| sort @timestamp desc
| limit 20

-or-

fields @timestamp, @message
| filter message like /Method completed with status: 504/
| sort @timestamp desc
| limit 20

Instrument X-Ray traces

If 504 errors keep happening continuously, determine where the Lambda function is spending its time. Instrument AWS X-Ray traces for Lambda functions based on the Lambda function runtime.

For Python:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()

For Node.js:

const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))

Note: You need to build a new deployment package after instrumenting X-Ray traces on a Lambda function.

Implement API retries

When a 504 error occurs and the request isn't found in Lambda, implement API retries on the client. The error might have resulted from a temporary network failure in API Gateway.

Check the Lambda function configuration

Make sure that your Lambda function has only API Gateway event-specific processing logic. This way, the Lambda function takes less time to run and can keep up with incoming events.

REST APIs have a default maximum integration timeout of 29 seconds. Therefore, make sure that the Lambda function's running duration is under 29 seconds.

If you have a use case where the application is time sensitive, see Set up asynchronous invocation of the backend Lambda function.