To monitor my Amazon EMR clusters, I run scripts that make API calls. The scripts return throttling or rate limit exceeded exceptions instead of the expected results.

Amazon EMR throttles API calls to maintain system stability. Throttling exceptions usually occur when you run monitoring scripts at regular intervals to check clusters for a parameter (for example, calling DescribeCluster every 60 seconds to check if a cluster has reached the WAITING state). The more clusters you have and the more monitoring scripts you run, the more likely you are to get throttling errors.

To prevent throttling errors:

  • Reduce the frequency of the API calls.
  • Stagger the intervals of the API calls so that they do not all run at once.
  • Implement exponential backoff when making API calls.

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2016-05-20

Updated: 2019-02-06