I periodically run a monitoring script to check the status of my EMR cluster. My script makes one or more EMR API calls but returns a throttling or rate limit exceeded exception instead of the expected results.
EMR API calls are limited to ensure system stability. Usually this scenario occurs when you run a monitoring script that checks the cluster for some parameter, such as calling DescribeCluster to check if a cluster has reached WAITING state, or calling ListSteps to check the progress of steps. If you run many clusters and many such scripts, you encounter throttling exceptions, especially if you make these calls at the same regular interval (for example, every 60 seconds).
Consider reducing the frequency of these calls or staggering the interval that you run your scripts so that they do not all run at once. Additionally, you should implement exponential backoff as described at Error Retries and Exponential Backoff in AWS when making AWS API calls.
EMR, request throttling, API, rate limit, backoff, HTTP Status Code: 400
Did this page help you? Yes | No
Back to the AWS Support Knowledge Center
Need help? Visit the AWS Support Center
Published: 2016-05-20
