My account is rate throttled (API throttled) when I perform bulk resource record operations on my Amazon Route 53 hosted zone. How can I fix this?
When you are performing bulk resource record operations for your hosted zone in Route 53, you might receive an HTTP 400 (Bad Request) error. A response header that contains a Code element with a value of Throttling and a Message element with a value of Rate exceeded indicates rate throttling.
Warning: Rate throttling happens when the number of API requests is greater than the hard limit of five requests per second (per account).
If Route 53 can't process the request before the next request for the same hosted zone arrives, it rejects subsequent requests with another HTTP 400 error. The response header contains a Code element with a value of PriorRequestNotComplete and a Message element with a value of the request was rejected because Route 53 was still processing a prior request.
Note: API calls by IAM users within the same account count towards global rate throttling for the account and affect API calls made from the AWS Management Console.
You can use the following methods to avoid rate throttling:
Group individual operations of the same type into one change batch operation to reduce API calls.
Note: UPSERT requests (update and insert) are counted twice.
For example, you could request to CREATE, DELETE, or LIST many records with one batch operation.
Using error retries and exponential backoff
Add error retries and exponential backoff to your Route 53 API calls.
For example, use a simple exponential backoff algorithm that retries the call in 2^i seconds, where i is the number of retries.
Randomizing start times
Randomize the start time for calling Route 53 APIs. Be sure that there aren't multiple applications executing the logic at the same time, because simultaneous requests can cause throttling.
Introducing "sleep time" between calls
If the code function calls to Route 53 APIs are consecutive, add "sleep time" in between two calls to minimize throttling risk.
Note: If your account is still rate throttled after using these troubleshooting methods, open a support case with Route 53 for assistance to locate the source IP address of the API calls that exceed the per second threshold. Then, you can shut off unnecessary sources, or use these troubleshooting methods to resolve the issue.