How do I troubleshoot high latency in my API Gateway requests that are integrated with Lambda?

Resolution

High latency must be addressed when an API endpoint that’s integrated with a Lambda function takes too long to send responses to a client. Review the metrics for API Gateway to identify the section of the request/response flow that’s causing high latency. After you determine the cause of high latency, you can work to reduce the delays.

Filter CloudWatch metrics to review latency metrics on the API

To identify the section of the request/response flow that’s causing high latency, first perform the following steps:

1. Observe the latency of the client after sending a request to the API.
2. After you note the overall latency, open the Amazon CloudWatch console. In the left navigation pane, choose Metrics, All metrics. In the metrics search box, enter APIGateway. From the search results, choose API Gateway, ApiId.
3. In the list of APIs, filter for the specified API by using the API ID or the API name. After filtering, check the IntegrationLatency and Latency check boxes.
   Note: The API ID and API name are available from the API Gateway console.
4. Open the Graphed metrics tab. For Statistic, choose Maximum. For Period, choose 1 minute. Above the graph, select the Custom time period. Choose the time frame during which the client experienced high latency.
5. Review both the IntegrationLatency and Latency metrics. Note the values and timestamps when these metrics have high values. The values can explain the cause for high latency.

Compare metrics to identify the cause of high latency

Continue to review the metrics related to the request/response flow to find the cause of high latency:

1. Compare the API Gateway Latency metric to the overall latency value observed at the client.
   For example, an API has a Latency metric with a Maximum value that’s approximately equal to the Max Latency value at the client. These values suggest that the maximum delay in the request/response flow is the time taken by API Gateway to process requests. API Gateway processing time includes the time taken to send requests to Lambda, wait for responses from Lambda, and send responses to a client.
2. Compare the IntegrationLatency metric with the Latency metric for the API.
   For example, the IntegrationLatency metric is approximately equal to the Latency metric. These values indicate that latency at the API is primarily caused by backend requests sent to Lambda that are taking longer to respond. The IntegrationLatency metric includes the time between API Gateway sending a request and API Gateway receiving a response from the backend
3. When the IntegrationLatency metric is low when compared to the Latency metric for the API, the backend response times are low. In this scenario, it takes longer to process the API requests or responses.
   For example, mapping templates configured in the API or an API Gateway Lambda authorizer both might create delays.
4. When the Latency metric for the API is much lower than the latency observed at the client, the route might be causing delays. Review the route between the client and API Gateway to confirm whether there are any intermediate endpoints that are adding delays.
   For example, private VPN connections or proxies might create delays.

View Lambda metrics to identify the cause of high IntegrationLatency

Finally, focus on the Lambda metrics related to the request/response flow to find the cause of high IntegrationLatency:

1. Check the Lambda function Duration metric to confirm whether the Lambda function’s execution time is longer. If the Lambda function’s execution time has increased, review the CloudWatch log to find the section of code that’s causing high latency. By default, Lambda functions log the START, END, and REPORT statements in CloudWatch logs. Add custom log statements at each logical section of the Lambda function code to get verbose CloudWatch logs.
2. If the Duration metric didn't change during the period of high latency at the client, determine if the initialization time increased. Initialization time in a Lambda function is the time taken to set up the execution environment to process a request. Requests that come from API Gateway might require a new environment for processing. This is set up through Lambda. Typically, the code that’s present outside the Lambda function handler runs during the initialization time. Code that takes longer to complete can cause delays for the overall response times to the client.
   Note: Initialization time is known as INIT or cold start.
3. Confirm whether there’s any increase in the initialization time’s Duration by verifying the report statements in the Lambda function logs. Initialization time that's high for some requests can cause an increase in the IntegrationLatency metric for API Gateway.

Related information

Working with metrics for HTTP APIs

Amazon API Gateway dimensions and metrics

Monitoring WebSocket API execution with CloudWatch metrics

Viewing metrics on the CloudWatch console

Lambda runtime environment lifecycle