AWS Security Blog
How to Use the REST API to Encrypt S3 Objects by Using AWS KMS
August 4, 2023: This blog post is out of date, and is in the process of being updated.
AWS Key Management Service (AWS KMS) allows you to use keys under your control to encrypt data at rest stored in Amazon S3. The two primary methods for implementing this encryption are server-side encryption (SSE) and client-side encryption (CSE). Each method offers multiple interfaces and API options to choose from. In this blog post, I will show you how to use the AWS REST APIs to secure S3 objects using KMS keys via SSE. Using the native REST APIs vs. the AWS SDKs may be a better option for you if you are looking to develop cross-platform code or want more control over API usage within your applications.
Overview of REST API call authentication
One critical aspect of using any AWS REST API call is the signature used to authenticate the request. This signature is vital if you wish to allow only authenticated users to access your data. There is an option to allow anonymous access that does not require a signature, but this makes your data publicly available to the world. For security, AWS recommends that you use only authenticated requests with a signature. This allows for verification of the requestor’s identity, in-transit data integrity protection, and protection against reuse of the signed portions of the request for replay attempts. I will describe the generic approach to signing AWS REST APIs, and then I will show how to use that technique to request that the S3 service encrypt objects on your behalf, using keys in your account that you manage using KMS.
Signatures are calculated by concatenating request elements to form a string such as a POST policy from an HTTP POST request. This is called the string to sign. Next, a signing key is used to calculate the keyed-hash method authentication code (HMAC) of the string to sign. In AWS Signature Version 4 signing, your secret access key is used to create the signing key. When S3 receives the request, it recreates the signature by using the authentication information provided to ensure a match. If it matches, it will accept/return your data, if not, an error is returned.
The following diagram shows an example of signature creation.
KMS default keys vs. customer-provided keys
The encryption key you use is an important factor in deciding to use SSE or CSE encryption. Some factors you should take into account when determining the best option for you are cost, flexibility, security, and maintenance. The most important factor, however, is access control. To have the proper security levels, you want to ensure that only permitted users are able to access the encryption keys needed to decrypt data. You control these access permissions in KMS by using policies.
With KMS, you also can see when, where, and by whom your customer managed keys (CMK) are used, because all API calls are logged by AWS CloudTrail. These logs provide you with full audit capabilities for your keys. There is also one default CMK for each account and service integrated with KMS. This key is referred to as the default service key and will always be listed with an alias of aws/[servicename] (for example, aws/s3) in your KMS console. This default service key is used if you choose not to create your own custom keys in KMS.
The first time you add a SSE-KMS encrypted object to a S3 bucket in a specific region, a default service key is created for you, which only allows users in your account the permission to use the key from S3. However, creating your own CMKs does give you more flexibility with access control, auditing, rotation, and deletion. It also allows you to use CMKs across multiple AWS services and from within your own applications.
Default service CMKs are assigned a default key policy that you cannot change. You can see the policy yourself by running the following AWS CLI command. (Replace the placeholder values with your own values.)
aws kms get-key-policy –key-id arn:aws:kms:region:111122223333:key/<32-char keyId>
The following policy example is the default key policy assigned to the default aws/s3 CMK. What this policy states is that all AWS Principals in the account can perform the Actions listed against S3 resources as long as they have the proper S3 permissions in their own IAM policies. This also gives permission to the account itself in order to pull the metadata of the key (account ID, ARN, description, and key state).
{
"Version": "2012-10-17",
"Id": "auto-s3-2",
"Statement": [{
"Sid": "Allow access through S3 for all principals in the account that are authorized to use S3",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "111122223333",
"kms:ViaService": "s3.region.amazonaws.com"
}
}
}, {
"Sid": "Allow direct access to key metadata in the account itself",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": [
"kms:Describe*",
"kms:Get*",
"kms:List*"
],
"Resource": "*"
}]
}
With a custom CMK, however, you can limit who can manage and use the key, and who is permitted to allow AWS resources to access KMS on your behalf. The following policy is an example of a policy assigned during custom CMK creation.
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-2",
"Statement": [{
"Sid": "Enable IAM user permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": "kms:*",
"Resource": "*"
}, {
"Sid": "Allow access for key administrators",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/IAMuser"
},
"Action": [
"kms:Describe*",
"kms:Put*",
"kms:Create*",
"kms:Update*",
"kms:Enable*",
"kms:Revoke*",
"kms:List*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*"
],
"Resource": "*"
}, {
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/IAMuser"
},
"Action": [
"kms:DescribeKey",
"kms:GenerateDataKey*",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:Decrypt"
],
"Resource": "*"
}, {
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/IAMuser"
},
"Action": [
"kms:ListGrants",
"kms:CreateGrant",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}]
}
With the preceding custom policy, you can be restrictive about the users, roles, and services that can have access to the key. This is one of the benefits of using a CMK in KMS. For more information about KMS key policies, see Key Policies.
Signature versions and request headers
When making all GET and PUT requests for an S3 object protected by KMS, you must use Signature Version 4 signing because Signature Version 2 signing is not supported. Signature Version 4 signing provides authentication using some or all of the following, depending on how you choose to sign the request:
- Verification of the identity of the requester.
- In-transit data protection.
- Protection against reuse of the signed portions of the request.
Request headers are a series of lines in the HTTP request that contain a name/value pair. These lines can be in any order and are used to pass information about the original request. Some examples of common request headers are: Content-Length, Content-Type, Date, Host, and x-amz-security-token. The following is an example of a basic request including headers:
GET /ObjectName HTTP/1.1 Host: BucketName.s3.amazonaws.com Date: date Authorization: authorization string
Some request headers are necessary for SSE encryption. The main required header is x-amz-server-side-encryption, which is used to request SSE-KMS with aws:kms used as the encryption format for object uploads. You must also use the x-amz-server-side-encryption-aws-kms-key-id header, because this specifies the ID of the KMS CMK you want to use. To add an extra layer of security that forces recipient to provide matching contextual information about the object, you would use x-amz-server-side-encryption-context. For more information, see the KMS documentation on Encryption Context and our blog post on the topic .
The following REST APIs accept the x-amz-server-side-encryption request header necessary to specify aws:kms as the encryption format for your S3 object uploads:
One of the easier ways to make REST API calls is to use a browser with a RESTful add-on. Two options to choose from are Postman – REST Client for Google Chrome and RESTClient for Mozilla Firefox. After you have installed the add-on, you can then use your browser to make REST API calls against your S3 objects. This is a fast and efficient way to access your data without having to program an application.
For this post, I am using Firefox and the RESTClient add-on, which you can see in the following screenshot.
How to Use the REST API to secure S3 objects with SSE-KMS
Now that I have covered the main components of S3 with SSE-KMS and making REST API calls, I can begin the process of using the REST API to secure S3 objects with SSE-KMS.
Step 1: Create an authorization signature using Signature Version 4 signing and the programming language of your choice
The authorization signature is used to authenticate the API requests to S3 so that your stored data is available only to authenticated users. The signature is created by combining data of the object and hash values created from your secret access key with other metadata.
Creating your signature can be done in various programming languages, but the examples I include in this post use JavaScript. The code snippets will show you how to create your string to sign, create the hash of your signing key, and provide the complete signature for API call authentication.
The following request does not use signing for authentication.
GET https://iam.amazonaws.com/?Action=ListUsers&Version=2010-05-08 HTTP/1.1 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Host: iam.amazonaws.com X-Amz-Date: 20150830T123600Z
The following request uses signing for authentication. Notice the presence of the Signature field:
GET https://iam.amazonaws.com/?Action=ListUsers&Version=2010-05-08 HTTP/1.1 Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7 content-type: application/x-www-form-urlencoded; charset=utf-8 host: iam.amazonaws.com x-amz-date: 20150830T123600Z
The JavaScript code I used to create the hashes for my signing key is forge.bundle.js. You can use the language your prefer, including Ruby, .NET, Python, and others. This will create the HMAC that is used to then create the final signature.
The “heavy lifter” of the group is the downloadable Presigned URL JavaScript code for Browser-based JavaScript POST script. This script creates your signature value by combining the hash values created by the forge.bundle.js script, the information for the file, and the KMS keys used for encryption. This signature value will be placed in the REST API call under the Authorization section in the body of your request.
You now have a completed signature to authenticate your REST API requests to S3. The next step focuses on creating the PUT REST API call by combining elements of the object and CMK for your request.
Step 2: Create a PUT REST API request
I will use a PUT request instead of POST because PUT is for creating or updating, and POST is only for creating. This request prevents duplicate objects. The PUT request places your data or object on your server for retrieval at a later date.
To upload data to S3 using the REST API, you need to create a PUT request. To do this, set the Method to PUT, specify the URL as an S3 URL (for example, https://s3-us-east-1.amazonaws.com), and then place the body of the request in the Body box. Be sure to include the x-amz headers to specify aws:kms as the method of encryption, and the key ID of the KMS CMK you wish to use for the actual data encryption. An example PUT REST API call follows.
PUT /example_image.jpg HTTP/1.1 Host: example-bucket.s3.amazonaws.com Accept: */* Authorization: 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7 Date: Wed, 28 May 2014 19:31:11 +0000 x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01
You should receive a response that resembles the following (the following screenshot shows how the response appears in the browser add-on).
HTTP/1.1 200 OK x-amz-id-2: 7qoYGN7uMuFuYS6m7a4lszH6in+hccE+4DXPmDZ7C9KqucjnZC1gI5mshai6fbMG x-amz-request-id: 06437EDD40C407C7 Date: Wed, 28 May 2014 19:31:11 +0000 x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01 ETag: "ae89237c20e759c5f479ece02c642f59"
Step 3: Create a GET REST API request
A GET request is used to recall data that has already been placed on a server. Think of it as opening a file or downloading an image. It is a way to retrieve information that was previously stored.
When you want to retrieve your data from S3, you must create a GET request. This request depends on your user having READ access to the objects that were uploaded. This will allow the user to open or download the requested object. Because this call is for an object encrypted with KMS, you will also need to provide the signature created previously for authorization. You follow the same steps as with the PUT request, but use the GET API instead. An example GET REST API call follows.
GET /example_image.jpg HTTP/1.1 Host: example-bucket.s3.amazonaws.com Accept: */* Authorization: 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7 Date: Wed, 28 May 2014 19:31:11 +0000 x-amz-server-side-encryption: aws:kms x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01
You should receive a response that resembles the following (the following screenshot shows how the response appears in the browser add-on).
HTTP/1.1 200 OK x-amz-id-2: ka5jRm8X3N12ZiY29Z989zg2tNSJPMcK+to7jNjxImXBbyChqc6tLAv+sau7Vjzh x-amz-request-id: 195157E3E073D3F9 Date: Wed, 28 May 2014 19:31:11 +0000 Last-Modified: Wed, 28 May 2014 19:31:11 +0000 ETag: "c12022c9a3c6d3a28d29d90933a2b096" x-amz-server-side-encryption: aws:kms x-amz-server-side-encryption-aws-kms-key-id:arn:aws:kms:us-east-1:111122223333:key/0695f802-503c-40n2-d17d-16d702f79f01
Summary
In this post, I have explained why you use signatures to authenticate REST API calls, what a signature includes, and how you create signatures. Using signatures for API authentication raises the security of your application. I also covered how SSE-KMS can add extra levels of security with access policies, encryption context, and request header specifications. Walking through how to add the signature value to your REST API requests should give you an idea of how easy this process can be—the most involved part of this process is the signature creation. From there, you are on your way to making REST API calls to quickly and efficiently place or retrieve your data securely.
If you have comments, submit them in the “Comments” section below. If you have questions, please start a new thread on the IAM forum.
– Tracy
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.