Manage your AWS KMS API request rates using Service Quotas and Amazon CloudWatch

AWS Key Management Service (KMS) publishes API usage metrics to Amazon CloudWatch and Service Quotas allowing you to both monitor and manage your AWS KMS API request rate quotas. This functionality helps you understand trends in your usage of AWS KMS and can help prevent API request throttling as you grow your use of AWS KMS.

When you surpass your AWS KMS API request rate quotas, you receive an error “You have exceeded the rate at which you may call KMS. Reduce the frequency of your calls.” Such errors can also be caused by an increased use of AWS services that encrypt your data under keys managed in AWS KMS. For example, if you are using Amazon Redshift Spectrum, you might encounter this error – “HTTP response error code: 503 Message: SlowDown. Please reduce your request rate for operations involving AWS KMS.” Historically, in order to understand how close to a request rate quota you were, you had to perform three tasks: (i) send AWS CloudTrail events generated by AWS KMS to Amazon CloudWatch Logs; (ii) write queries in Amazon CloudWatch Logs Insights to track your API request usage; and (iii) submit an AWS Support case to request a quota increase. Now, you can view your AWS KMS API usage and request quota increases within the AWS Service Quotas console itself without doing any special configuration.

In this post, we will show you how to 1) view your KMS API utilization within Service Quotas 2) create a CloudWatch Alarm that alerts you to an approaching quota so you can request quota increases before you are throttled.

View your AWS KMS API utilization

Background

API utilization is the percentage rate at which you are calling a particular API compared to that API’s request rate quota in your account. For AWS KMS, the default request rate for cryptographic operations using symmetric keys is 10,000 requests per second in 6 specific AWS Regions*, aggregated across all requesting clients in an account. AWS KMS aggregates your API requests every minute and sends it to CloudWatch, where it is consumed by AWS Service Quotas for you to see. Because quota usage is aggregated by the minute, your effective quota would be 600,000 requests per minute.

*See Request Quotas for Each AWS KMS API Operation for the specific quotas in the AWS Region in which you operate.

Scenario

Imagine that all the applications in your account using AWS KMS collectively made 100,000 requests to the Decrypt API, 100,000 requests to the GenerateDataKey API, and 100,000 requests to the Encrypt API in a minute. AWS KMS sends a count of 300,000 requests to Amazon CloudWatch for that particular minute. Your utilization for that minute will be 50% of your quota (300,000 divided by 600,000, which is 60 seconds times your quota of 10,000 requests per second). Within the Service Quotas console, you can view utilization across several time frames, from the most recent hour up to a week.

Here are the steps to view your AWS KMS API Utilization within Service Quotas:

1. Sign in to the AWS Management Console.
2. Click on “Services” dropdown on the top left corner and search for “Service Quotas” and select it from the dropdown.
3. Click on the AWS Key Management Service (AWS KMS) tile on the Service Quotas dashboard.
4. Search for “symmetric” and click on the link for “Cryptographic operations (symmetric) request rate”.
5. The Monitoring section will display the combined utilization percentage for the following APIs – Decrypt, Encrypt, GenerateDataKey, GenerateDataKeyWithoutPlaintext, GenerateRandom, and ReEncrypt. All these APIs are grouped under the shared “Cryptographic operations (symmetric) request rate”.
6. Adjust the graph to view the utilization trend over a week by selecting “1w” from the top right corner of the graph.

You can view the utilization for any of the other available AWS KMS APIs from the Service Quotas dashboard in a similar fashion.

The API utilization provides you the overall trend of your API usage. Because the requests sent from AWS KMS are aggregated per minute, you could still experience throttling errors at a less than 100% utilization, especially if your usage is spiky and if you do not have exponential back off built into your applications’ error handling logic. For example, you might have surpassed the requests per second quota between the 12th second and the 15th second of the minute, but you were below the quota for the other 57 seconds of that minute.
 

The utilization shown is across your entire AWS account in a given region, so if you are introducing a new application, you can monitor and see how it impacts your overall utilization. If you need a request rate quota increase before deploying your new application to production, you can request a quota increase at the top right portion of the Details section of the AWS Service Quotas page.

Create a CloudWatch Alarm

In the previous section we described how you can view historical utilization of API request rates from the Monitoring section of the AWS Service Quotas console. What if you want to be alerted when you have reached a predetermined utilization percentage so you can request a quota increase before you begin to experience extended throttling?

Here are the steps to do so:

1. Click on the API of your interest from the Service Quotas console. In this example, let’s select Cryptographic operations (symmetric) request rate.
2. In the Amazon CloudWatch alarms section (under the Monitoring section), click Create on the right hand corner.
3. From the Alarm threshold dropdown select “80% of applied quota value”.
4. Enter “80threshold” as the Alarm name and click the orange Create button on the right side.
5. Click on the “80threshold” link that now appears in the table. A new browser window will appear that takes you to the Amazon CloudWatch console.
6. Click Edit on the top right corner.
7. Leave all the default values selected on the Specify metrics and condition page and click Next on the bottom right.
8. Click Add notification and select Create new topic under the Select an SNS topic section. Enter “SNS-Topic” as the topic name. Add your email address to receive notifications when the alarm is set. Click Create topic.
9. Click Update alarm.
10. Confirm your SNS subscription by clicking on View SNS Subscriptions.
     
    

 

11. Select your email address endpoint and click Request confirmation.
12. You will receive an email to confirm your subscription. Once you confirm the subscription, you are all set to receive email notifications on the new alarm.
     
    

Here are more details on creating CloudWatch alarms if you want to make additional modifications to your alarms. We recommend 80% as a good threshold to set your alarm to begin with. When you are testing a new application, you can start with this threshold and run your application for a period of time and monitor its utilization. When an alarm fires, you can you can proactively request a quota increase at the top right portion of the Details section of the AWS Service Quotas page.

Conclusion

We’ve explored how to view your AWS KMS API request usage, how to add alarms on the most critical items in your application’s use of AWS KMS, and how to request quota increases. These items provide visibility and control over how your applications interact with AWS KMS.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread in the AWS Key Management Service forums.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.