How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?
Last updated: 2021-04-14
My AWS Lambda function is producing "Rate exceeded" and 429 "TooManyRequestsException" errors. Why is my function throttled, and how do I resolve the errors?
Lambda functions are sometimes throttled to protect your resources and downstream applications. Even though Lambda automatically scales to accommodate incoming traffic, your function can still be throttled for various reasons.
To troubleshoot Lambda throttling issues, including Rate exceeded and TooManyRequestsException errors, do the following:
Verify what resource is throttled
Throttle errors might not be from your Lambda function. Throttles can also occur on API calls during your function's invocation.
To confirm what resource is throttled, do the following:
If there are no Lambda Throttles metrics, then the throttling is happening on API calls in your Lambda function code.
Check your function code for any throttled API calls
If certain API calls are throttled, make sure that you use exponential backoff in your code to retry the API calls.
Note: If you determine that you need a higher transactions per second (TPS) quota for an API call, you can request a service quota increase. Not all quotas are adjustable.
Check your function's concurrency metrics
Review your Lambda metrics in Amazon CloudWatch
Check the ConcurrentExecutions metric for your function in the AWS Region where you see throttling.
Compare the ConcurrentExecutions metric with the Throttles metric for the same timestamp
View the Maximum statistic for ConcurrentExecutions and the Sum statistic for Throttles. See if the maximum ConcurrentExecutions are close to your account-level concurrency quota in the Region, along with corresponding data points in the Throttles graph.
Check if you're exceeding the initial burst concurrency quota for a particular AWS Region
On the Metrics page for Lambda in the CloudWatch console, reduce the graph's time range to one minute. If you're limited by burst scaling, then you see a spike of Throttles that corresponds to a stair-step pattern of ConcurrentExecutions on the graph.
Note: To work around burst concurrency limits, you can configure provisioned concurrency.
Check for spikes in Duration metrics for your function
Concurrency depends on function duration. If your function code is taking too long to complete, then there might not be enough compute resources.
Note: Changing the memory setting can affect the charges that you incur for execution time.
If your function is in an Amazon Virtual Private Cloud (Amazon VPC), see How do I give internet access to a Lambda function that's connected to an Amazon VPC?
Check for an increase in Error metrics for your function
Increased errors can lead to retries and cause an overall increase in invocations. Increased invocations can lead to an increase in concurrency. Use CloudWatch Logs to identify and eliminate errors, and have your function code handle exceptions.
Note: Your function can also be throttled based on invocation requests per Region (requests per second), which is 10 times your concurrent executions quota.
Configure reserved concurrency
Confirm that you've configured reserved concurrency on your function
Note: If a function is configured to have zero reserved concurrency, then the function is throttled because it can't process any events. Make sure that you increase the value to a number greater than zero.
Review the Maximum statistic in CloudWatch for your function
See if the function metric hits the maximum value for the ConcurrentExecutions metric at any point.
Increase the reserved concurrency for your function to a concurrency value that keeps the function from being throttled
Use exponential backoff in your app
As a best practice, retry throttled requests by using exponential backoff in your application that's calling your Lambda function.
Use a dead-letter queue
For asynchronous event sources, such as Amazon Simple Storage Service (Amazon S3) and Amazon CloudWatch Events, configure your function with a dead-letter queue (DLQ). The DLQ catches any events that are discarded due to constant throttles and can protect your data if you're seeing significant throttling.
Note: For Amazon Simple Queue Service (Amazon SQS) event sources, you must configure the DLQ on the Amazon SQS queue.