Enhance savings for read-heavy workloads with Amazon S3 Bucket Keys

Organizations continue to grow their data lakes in the cloud as they build out new and innovative analytics, machine-learning, and generative AI workloads. At the same time, these workloads often access data that requires compliance with stringent data security and privacy standards. These compliance frameworks typically specify additional requirements for encryption at-rest, which leads customers to key management services that offer additional customization and control. In the cloud, these key management services are priced based on their usage leading customers to look for cost optimization solutions.

For data stored in Amazon Simple Storage Service (Amazon S3), customers use AWS Key Management Service (AWS KMS) to help adhere to applicable compliance requirements. Server-side encryption with AWS KMS (SSE-KMS) incurs a per-request AWS KMS charge for encryption and decryption cryptographic operations. As customers scale their workloads to millions or billions of objects, this increases their AWS KMS costs. Customers experience this acutely in heavy read/write applications, where data is accessed frequently. To manage these costs, customers often enable S3 Bucket Keys to reduce their AWS KMS requests, but observed lower savings than expected when using services that create multiple IAM sessions, like Amazon EMR or AWS Identity and Access Management (AWS IAM) AssumeRole actions.

In this post, we discuss how S3 Bucket Keys can help reduce your KMS costs and explore a recent enhancement to improve your savings especially for read-heavy workloads. This enhancement allows IAM assume role sessions to share a bucket-level key when derived from the same IAM role and matching scoped down policy. With this new improvement, you can realize even greater savings on your AWS KMS request costs when using S3 Bucket Keys.

Amazon S3 Bucket Keys

In 2020, Amazon S3 launched S3 Bucket Keys, which help you reduce the cost of server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). AWS customers using S3 Bucket Keys have saved over $80 million on AWS Key Management Service requests since launch.

When using S3 Bucket Keys, instead of an individual KMS data key for each KMS encrypted object, a bucket-level key is generated by KMS. S3 uses this bucket-level key to create unique data keys for objects in a bucket, reducing the need for additional KMS requests to complete encryption operations. This results in the reduction of request traffic from S3 to KMS, allowing you to access encrypted objects in S3 at a fraction of the previous cost.

New enhancements for read-heavy workloads

Although S3 Bucket Keys help customers across the board save on their KMS requests, customers that used temporary credentials, like AWS Security Token Service (AWS STS) AssumeRole actions, experienced less savings. These short-lived credentials called a new bucket-level key for each IAM role session, which reduced savings with S3 Bucket Keys. This issue was compounded for customers using data analytics services, like Amazon EMR, that create many IAM sessions even though all of the sessions belong to the same IAM role with identical scoped down policies.

To improve customer savings, Amazon S3 deployed an update to allow a bucket-level key to be shared across all IAM sessions that are derived from the same IAM role and identical scoped down IAM policy. Amazon S3 continues to treat callers as different requesters when they use different roles or accounts, or the same role with differently scoped policies. This enhancement allows customers to improve their AWS KMS savings with Bucket Keys when using services like an Amazon EMR cluster composed of multiple Amazon EC2 instances or STS AssumeRole actions using the same role with a matching scoped down policy. This improvement requires no changes to your applications and takes effect on both new and existing objects encrypted with S3 Bucket Keys.

Cost savings improvements

When analyzing customer-specific analytics workloads, the Amazon S3 team observed KMS requests savings to increase from 55% to as much as 99%. Here are some quotes from customers seeing similar savings:

“FINRA started using S3 Bucket Keys on our S3 data as soon as the feature launched in 2020 to help us save on AWS KMS request costs while still adhering to our encryption compliance requirements. With the recent enhancement to S3 Bucket Keys, we’ve seen our KMS costs decrease by as much as 50% as more of our requests could use Bucket Keys.”

“Pomelo is leveraging AWS to gain new business insights through our data lake workloads. We were initially using SSE-KMS to encrypt all of our data, but our read-heavy access patterns quickly generated high KMS request costs. Implementing S3 Bucket Keys for both our new and existing data helped us reduce our KMS request costs by 95%.”

Conclusion

Many customers have compliance standards requiring they control their encryption keys; these customers use SSE-KMS with customer-managed keys to adhere to those requirements. To help these customers save on AWS KMS requests when using SSE-KMS encryption, Amazon S3 offers S3 Bucket Keys. Enabling S3 Bucket Keys is a recommended best practice for customers with compliance requirements for data encryption. This feature can help you save on your AWS KMS bill while still helping you comply with data encryption standards.

In this blog post, we reviewed a new enhancement for S3 Bucket Keys when using temporary credentials to help you automatically save more on your AWS KMS bill. With this update, AssumeRole sessions using the same IAM role and an identical scoped down policy can share a bucket-level key. This enhancement is available automatically without needing to change your applications and can improve savings for both reads and writes on objects encrypted with S3 Bucket Keys.

Thanks for reading this blog. If you have any comments or questions, don’t hesitate to leave them in the comments section.