Reducing AWS Key Management Service costs by up to 99% with Amazon S3 Bucket Keys

Customers across many industries face increasingly stringent audit and compliance requirements on data security and privacy. Certain compliance frameworks, such as FISMA, FEDRAMP, PCI DSS, and SOC 2, have specific regulatory standards for validating the security of systems. A common requirement for these compliance frameworks is more rigorous encryption standards for data-at-rest, where organizations must encrypt data with an encryption key that they control. To satisfy these requirements on premises, customers must dedicate resources to maintain cryptographic modules. In the cloud, customers use encryption offerings that are priced based on their usage.

Amazon Simple Storage Service (Amazon S3) encrypts all new objects with Amazon S3-managed server-side encryption (SSE-S3), unless the customer selects another type of server-side encryption. SSE-S3 satisfies the security requirements for many Amazon S3 customers, but customers adhering to some compliance frameworks may need greater control over their encryption keys. These customers can rely on AWS Key Management Service (AWS KMS) and server-side encryption with AWS KMS (SSE-KMS). SSE-KMS incurs both a per-request AWS KMS charge and a monthly storage charge per AWS KMS key. As customers scale their workloads to millions or billions of objects, their AWS KMS costs can increase. Customers experience this acutely in heavy read/write applications, where customer data is accessed at a high frequency. To help customers reduce this cost, Amazon S3 introduced S3 Bucket Keys. In fact, S3 Bucket Keys can reduce AWS KMS costs by up to 99%, and have saved customers over $80M since they were launched at the end of 2020.

In this post, I show you how to require SSE-KMS for all objects written to the bucket, how to enable S3 Bucket Keys on your buckets, how to use S3 Bucket Keys in a multi-tenant bucket, and how to evaluate your existing objects encrypted via object level SSE-KMS. Using S3 Bucket Keys helps you save on your AWS KMS bill while still complying with security requirements to encrypt your data with SSE-KMS.

Background

SSE-KMS provides customers control over their encryption keys and the ability to audit key usage. These additional controls are very important for enterprise customers that must demonstrate to auditors the ownership of their encryption keys, control of key rotation schedules, and management of encryption key permissions. When an object is encrypted with SSE-KMS, AWS KMS receives an encryption request from Amazon S3, and sends back a unique data key to encrypt the object.

With S3 Bucket Keys, instead of an individual call to AWS KMS to encrypt each object, a time-limited bucket-level key is generated by AWS KMS. Amazon S3 uses this bucket-level key to create unique data keys to encrypt objects in a bucket, avoiding the need for additional AWS KMS requests to complete cryptographic operations. S3 Bucket Keys reduce request traffic from Amazon S3 to AWS KMS, allowing you to access encrypted objects in Amazon S3 at a fraction of the previous cost.

Prerequisites

To follow along with the blog post, your AWS Identity and Access Management (IAM) principal (user or role) needs the permissions to create buckets and to use AWS KMS keys.

Minimum IAM permissions to use Bucket Keys
{
   "Version":"2012-10-17",
   "Statement":[
{
   "Sid": "Permissions to use Bucket Keys",
   "Effect":"Allow",
   "Principal": {"AWS": "arn:aws:iam::111122223333:user/willcavin"},
         "Action":[
         "s3:PutObject",
         "s3:GetObject",
         "s3:CreateBucket",
         "kms:Decrypt",
         "kms:GenerateDataKey"
         ],
         "Resource":"arn:aws:s3:::willcavinbucket/*"
      }
   ]
}

Figure 1: Example IAM permissions to use S3 Bucket Keys

Require SSE-KMS for all objects written to the bucket

Customers with compliance requirements often want to require encryption with SSE-KMS for all objects uploaded to their S3 buckets. This can be accomplished through S3 Bucket Policies. The following bucket policy denies any write to the bucket that is not encrypted with SSE-KMS. You can successfully upload objects to a bucket with this policy by adding the encryption request header on your PutObject request, which specifies SSE-KMS. Alternatively, you can set a default bucket encryption configuration on your bucket to automatically encrypt new objects with SSE-KMS, even if no encryption header is added to upload requests by your clients. This bucket policy combined with enabling S3 Bucket Keys on your default bucket encryption configuration verifies objects uploaded use S3 Bucket Keys unless you specifically disable S3 Bucket Keys on the PutObject request.

 {
 "Version": "2012-10-17",
 "Id": "PutObjPolicy",
 "Statement": [{
 		"Sid": "DenyObjectsThatAreNotSSEKMS",
 		"Effect": "Deny",
 		"Principal": "*",
 		"Action": "s3:PutObject",
 		"Resource": "arn:aws:s3:::example-bucket/*",
 		"Condition": {
 		"Null": {
 		"s3:x-amz-server-side-encryption-aws-kms-key-id": "true"
           }
      }
   }]
 }

Figure 2: Example IAM permissions to enable S3 Bucket Keys

Enabling S3 Bucket Keys on your bucket

S3 Bucket Keys can be enabled through the Amazon S3 console by editing the default encryption on your bucket. After selecting your bucket, choose Properties. Under Default encryption, choose Edit.

Under Encryption key type, choose AWS Key Management Service key (SSE-KMS). Under AWS KMS Key, choose your preferred AWS KMS key. Under Bucket Key, choose Enable. Choose Save changes.

Figure 3: Example default encryption configuration to enable S3 Bucket Keys

Using the AWS CLI

aws s3api put-bucket-encryption --bucket <bucket-name> --server-side-encryption-configuration '{
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "<KMS-Key-ARN>"
                },
                "BucketKeyEnabled": true
            }
        ]
    }'

Note that Amazon S3 enables S3 Bucket Keys for new objects added to your bucket. Existing objects do not use S3 Bucket Keys.

Using S3 Bucket Keys in a multi-tenancy environment

Many customers use S3 buckets in multi-tenant scenarios where they may have multiple teams or end-customers uploading objects to a shared bucket for analytics or data processing. S3 Bucket Keys were designed to work in this model. You can simply enable S3 Bucket Keys in your bucket default encryption configuration and have your end-customers add their preferred customer-managed AWS KMS key to their Amazon S3 request headers. This allows end-customers to encrypt objects with their customer-managed AWS KMS key and provide multi-tenant bucket owners the ability to enable S3 Bucket Keys for objects uploaded to their bucket.

In this construct, each tenant shares their customer-managed AWS KMS key with you so you can perform your services (data analytics, data processing, etc.). If a tenant decides they no longer want you to access their data, then they simply remove your access to their key. This model maintains key material confidentiality while providing access to the appropriate parties.

Figure 4: Example multi-tenant bucket using AWS KMS keys

Evaluating read-heavy buckets with SSE-KMS objects

Since the launch of S3 Bucket Keys, many customers have re-encrypted their existing data to adopt Bucket Keys, and lower their AWS KMS costs. If you have a read-heavy bucket with objects encrypted via object-level SSE-KMS, then you should carefully evaluate your access patterns to see if this makes sense for your applications. You can use S3 Storage Lens to identify AWS KMS requests across your AWS Organization, then zoom in on the accounts and buckets with the most AWS KMS requests. If you typically only access data for a short period of time after it is created, then we recommend simply letting your objects age out and using S3 Bucket Keys for all new objects. If your data is very read-heavy over a longer period of time, perhaps years, then it may make sense to re-encrypt your existing object-level SSE-KMS objects with S3 Bucket Keys.

To encrypt your existing Amazon S3 objects with S3 Bucket Keys, you can use Amazon S3 Batch Operations. You provide S3 Batch Operations with a list of objects and Batch Operations executes the specified operation on each of them. You can use the S3 Batch Operations Copy operation to copy existing SSE-KMS objects and write them back to the same bucket to enable S3 Bucket Keys on these objects. A single Batch Operations job can perform the specified operation on billions of objects. This post provides a great example for how you can use an AWS Lambda function to retrieve the SSE-KMS key ID for each object and enable S3 Bucket Keys by copying the objects in place with the appropriate customer-managed key.

Things to know

Many customers often want to quickly audit their bucket default encryption configuration across their accounts for buckets without S3 Bucket Key enabled. This post details how you can quickly identify and remediate buckets without this cost savings feature enabled.

Changes to note before enabling an S3 Bucket Key

If your existing IAM policies or AWS KMS key policies use your object Amazon Resource Name (ARN) as the encryption context to refine or limit access to your AWS KMS key, then these policies won’t work with S3 Bucket Keys. S3 Bucket Keys use the bucket ARN as the encryption context. Before you enable an S3 Bucket Key, update your IAM policies or AWS KMS key policies to use your bucket ARN as the encryption context.

After you enable S3 Bucket Keys, you see fewer AWS KMS CloudTrail events for SSE-KMS objects in your logs, as fewer calls are made to AWS KMS. Additionally, your AWS KMS CloudTrail events log your bucket ARN instead of your object ARN.

Using the Copy API

You can use the Copy operation to re-encrypt an existing object. Note that the Copy operation changes the Last Modified Date for the object, which resets the object’s Lifecycle eligibility if you have Lifecycle rules in place. Optionally, you can add a new Object Tag during the Copy operation to retroactively identify the re-encrypted objects. Then, you can create a new Lifecycle rule to transition these tagged objects to colder storage or expire them at your discretion. Additionally, any object greater than 5 GB must be copied using the Multipart Upload API, as the Copy API does not support objects greater than 5 GB.

Cleaning up

If you created new resources to test out the suggestions shared in this post, then delete your customer-managed AWS KMS keys and any objects you added to your buckets.

Conclusion

Many customers have compliance requirements mandating customer’s control their encryption key, these customers use SSE-KMS with customer-managed keys to adhere to these regulatory standards. In this blog post, we learned how you can enforce SSE-KMS encryption at the bucket-level with S3 Bucket Policies. We also learned how you can enable S3 Bucket Keys in your default bucket encryption configuration to easily take advantage of this cost-saving feature. We discussed how multi-tenant bucket owners can enable S3 Bucket Keys while still letting their customers use their own customer-managed KMS keys. Finally, we reviewed some important access pattern considerations before re-encrypting existing SSE-KMS objects to use S3 Bucket Keys.

Enabling S3 Bucket Keys is a recommended best practice for customers with compliance requirements for data encryption. This feature provides savings on your AWS KMS bill while still helping you comply with data encryption standards. You now have the knowledge and tools to maximize your savings, using S3 Bucket Keys.

Thanks for reading this blog, if you have any comment or questions, don’t hesitate to leave them in the comments section.