How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?

Last updated: 2021-04-14

My AWS Lambda function is producing "Rate exceeded" and 429 "TooManyRequestsException" errors. Why is my function throttled, and how do I resolve the errors?

Resolution

Lambda functions are sometimes throttled to protect your resources and downstream applications. Even though Lambda automatically scales to accommodate incoming traffic, your function can still be throttled for various reasons.

To troubleshoot Lambda throttling issues, including Rate exceeded and TooManyRequestsException errors, do the following:

Verify what resource is throttled

Throttle errors might not be from your Lambda function. Throttles can also occur on API calls during your function's invocation.

To confirm what resource is throttled, do the following:

Check your Amazon CloudWatch Logs to see if there are throttling errors, but no corresponding data points in the Lambda Throttles metrics

If there are no Lambda Throttles metrics, then the throttling is happening on API calls in your Lambda function code.

Check your function code for any throttled API calls

If certain API calls are throttled, make sure that you use exponential backoff in your code to retry the API calls.

Note: If you determine that you need a higher transactions per second (TPS) quota for an API call, you can request a service quota increase. Not all quotas are adjustable.

Check your function's concurrency metrics

Review your Lambda metrics in Amazon CloudWatch

Check the ConcurrentExecutions metric for your function in the AWS Region where you see throttling.

Compare the ConcurrentExecutions metric with the Throttles metric for the same timestamp

View the Maximum statistic for ConcurrentExecutions and the Sum statistic for Throttles. See if the maximum ConcurrentExecutions are close to your account-level concurrency quota in the Region, along with corresponding data points in the Throttles graph.

Check if you're exceeding the initial burst concurrency quota for a particular AWS Region

On the Metrics page for Lambda in the CloudWatch console, reduce the graph's time range to one minute. If you're limited by burst scaling, then you see a spike of Throttles that corresponds to a stair-step pattern of ConcurrentExecutions on the graph.

Note: To work around burst concurrency limits, you can configure provisioned concurrency.

Check for spikes in Duration metrics for your function

Concurrency depends on function duration. If your function code is taking too long to complete, then there might not be enough compute resources.

Try increasing the function's memory setting. Then, use AWS X-Ray and CloudWatch Logs to isolate the cause of duration increases.

Note: Changing the memory setting can affect the charges that you incur for execution time.

If your function is in an Amazon Virtual Private Cloud (Amazon VPC), see How do I give internet access to a Lambda function that's connected to an Amazon VPC?

Check for an increase in Error metrics for your function

Increased errors can lead to retries and cause an overall increase in invocations. Increased invocations can lead to an increase in concurrency. Use CloudWatch Logs to identify and eliminate errors, and have your function code handle exceptions.

Note: Your function can also be throttled based on invocation requests per Region (requests per second), which is 10 times your concurrent executions quota.

Configure reserved concurrency

Confirm that you've configured reserved concurrency on your function

Check the setting using the Lambda console, or by calling the GetFunction API.

Note: If a function is configured to have zero reserved concurrency, then the function is throttled because it can't process any events. Make sure that you increase the value to a number greater than zero.

Review the Maximum statistic in CloudWatch for your function

See if the function metric hits the maximum value for the ConcurrentExecutions metric at any point.

Increase the reserved concurrency for your function to a concurrency value that keeps the function from being throttled

Change the setting using the Lambda console, or by calling the PutFunctionConcurrency API.

Use exponential backoff in your app

As a best practice, retry throttled requests by using exponential backoff in your application that's calling your Lambda function.

Use a dead-letter queue

For asynchronous event sources, such as Amazon Simple Storage Service (Amazon S3) and Amazon CloudWatch Events, configure your function with a dead-letter queue (DLQ). The DLQ catches any events that are discarded due to constant throttles and can protect your data if you're seeing significant throttling.

Note: For Amazon Simple Queue Service (Amazon SQS) event sources, you must configure the DLQ on the Amazon SQS queue.

Request a service quota increase

If you determine that your workload requires a higher service quota for concurrent executions, request a service quota increase using the Service quotas console.