AWS Security Blog

New AWS Encryption SDK for Python Simplifies Multiple Master Key Encryption

September 8, 2021: AWS KMS is replacing the term customer master key (CMK) with AWS KMS key and KMS key. The concept has not changed. To prevent breaking changes, AWS KMS is keeping some variations of this term. More info.

The AWS Cryptography team is happy to announce a Python implementation of the AWS Encryption SDK. This new SDK helps manage data keys for you, and it simplifies the process of encrypting data under multiple master keys. As a result, this new SDK allows you to focus on the code that drives your business forward. It also provides a framework you can easily extend to ensure that you have a cryptographic library that is configured to match and enforce your standards. The SDK also includes ready-to-use examples. If you are a Java developer, you can refer to this blog post to see specific Java examples for the SDK.

In this blog post, I show you how you can use the AWS Encryption SDK to simplify the process of encrypting data and how to protect your encryption keys in ways that help improve application availability by not tying you to a single region or key management solution.

How does the AWS Encryption SDK help me?

Developers using encryption often face three problems:

  1. How do I correctly generate and use a data key to encrypt data?
  2. How do I protect the data key after it has been used?
  3. How do I store the data key and ciphertext in a portable manner?

The library provided in the AWS Encryption SDK addresses the first problem by implementing the low-level envelope encryption details transparently using the cryptographic provider available in your development environment. The library helps address the second problem by providing intuitive interfaces to let you choose how you want to generate data keys and the master keys or key-encrypting keys that will protect data keys. Developers can then focus on the core of the application they are building instead of on the complexities of encryption. The ciphertext addresses the third problem, as described later in this post.

The AWS Encryption SDK defines a carefully designed and reviewed ciphertext data format that supports multiple secure algorithm combinations (with room for future expansion) and has no limits on the types or algorithms of the master keys. The ciphertext output of clients (created with the SDK) is a single binary blob that contains your encrypted message and one or more copies of the data key, as encrypted by each master key referenced in the encryption request. This single ciphertext data format for envelope-encrypted data makes it easier to ensure the data key has the same durability and availability properties as the encrypted message itself.

The AWS Encryption SDK provides production-ready reference implementations in Java and Python with direct support for key providers such as AWS Key Management Service (KMS). The Java implementation also supports the Java Cryptography Architecture (JCA/JCE) natively, which includes support for AWS CloudHSM and other PKCS #11 devices. The standard ciphertext data format the AWS Encryption SDK defines means that you can use combinations of the Java and Python clients for encryption and decryption as long as they each have access to the key provider that manages the correct master key used to encrypt the data key.

Let’s look at how you can use the AWS Encryption SDK to simplify the process of encrypting data and how to protect your data keys in ways that help improve application availability by not tying you to a single region or key management solution.

Example 1: Encrypting application secrets under multiple regional KMS master keys for high availability

Many customers want to build systems that not only span multiple Availability Zones, but also multiple regions. You cannot share AWS KMS keys across regions. However, with envelope encryption, you can encrypt the data key with multiple KMS keys in different regions. Applications running in each region can use the local KMS endpoint to decrypt the ciphertext for faster and more reliable access.

For the examples in this post, I will assume that I am running on Amazon EC2 instances configured with IAM roles for EC2. This enables me to avoid credential management and take advantage of built-in logic that routes requests to the nearest endpoints. These examples also assume that the latest version of the AWS SDK for Python (different from the AWS Encryption SDK) is available.

The encryption logic has a simple high-level design. Using provided parameters, I get the master keys and use them to encrypt some provided data, as shown in the following code example. I will define how to construct the multi-region KMS key provider next.

import aws_encryption_sdk


def encrypt_data(plaintext):
    # Get all the master keys needed
    key_provider = build_multiregion_kms_master_key_provider()

    # Encrypt the provided data
    ciphertext, header = aws_encryption_sdk.encrypt(
        source=plaintext,
        key_provider=key_provider
    )
    return ciphertext

Create a master key provider containing multiple master keys

The following code example shows how you can encrypt data under KMS keys in three US regions: us-east-1, us-west-1, and us-west-2. The example assumes that you have already set up the KMS keys and created an alias named alias/exampleKey in each region for each KMS keys. For more information about creating KMS keys and aliases, see Creating Keys in the AWS KMS documentation.

This example creates a single KMSMasterKeyProvider to which all KMS keys are added. The KMSMasterKeyProvider handles interacting with KMS keys in multiple regions. Note that the first master key added to the KMSMasterKeyProvider is the one used to generate the new data key, and the other master keys are used to encrypt the new data key.

import aws_encryption_sdk
import boto3


def build_multiregion_kms_master_key_provider():
    regions = ('us-east-1', 'us-west-1', 'us-west-2')
    alias = 'alias/exampleKey'
    arn_template = 'arn:aws:kms:{region}:{account_id}:{alias}'

    # Create AWS KMS master key provider
    kms_master_key_provider = aws_encryption_sdk.KMSMasterKeyProvider()

    # Find your AWS account ID
    account_id = boto3.client('sts').get_caller_identity()['Account']

    # Add the KMS alias in each region to the master key provider
    for region in regions:
        kms_master_key_provider.add_master_key(arn_template.format(
            region=region,
            account_id=account_id,
            alias=alias
        ))
    return kms_master_key_provider

The logic to construct a master key provider could be built once by your central security team and then reused across your company to both simplify development and ensure that all encrypted data meets corporate standards.

Encrypt the data

The data you encrypt can come from anywhere and you can distribute it however you like. In the following code example, I read a file from disk and write out an encrypted copy. The AWS Encryption SDK provides a stream interface that behaves as a standard Python stream context manager to make this easy.

import aws_encryption_sdk
import boto3


def encrypt_file(input_filename, output_filename):
    # Get all the master keys needed
    key_provider = build_multiregion_kms_master_key_provider()

    # Open the files for reading and writing
    with open(input_filename, 'rb') as infile,\
            open(output_filename, 'wb') as outfile:
        # Encrypt the file
        with aws_encryption_sdk.stream(
            mode='e',
            source=infile,
            key_provider=key_provider
        ) as encryptor:
            for chunk in encryptor:
                outfile.write(chunk)

This file could contain, for example, secret application configuration data (such as passwords, certificates, and the like) that is then sent to EC2 instances as EC2 user data upon launch.

Decrypt the data

The following code example decrypts the contents of the EC2 user data and writes it to the specified file. The KMSMasterKeyProvider  defaults to using KMS in the local region, so decryption proceeds quickly without cross-region calls.

from botocore.vendored import requests


def decrypt_user_data(output_filename):
    # Create a master key provider that points to the local KMS stack
    kms_key_provider = aws_encryption_sdk.KMSMasterKeyProvider()

    # Read the user data
    user_data = requests.get('http://169.254.169.254/latest/user-data/').content
    # Open a stream to write out the decrypted file
    # Decrypt the userdata and write the plaintext into the file
    with open(output_filename, 'wb') as outfile,\
            aws_encryption_sdk.stream(
                mode='d',
                source=user_data,
                key_provider=kms_key_provider
            ) as decryptor:
        for chunk in decryptor:
            outfile.write(chunk)

Congratulations! You have just encrypted data under master keys in multiple regions and have code that will always decrypt the data by using the local KMS stack. This gives you higher availability and lower latency for decryption, while still only needing to manage a single ciphertext.

Example 2: Encrypting application secrets under master keys from different providers for escrow and portability

Another reason why you might want to encrypt data under multiple master keys is to avoid relying on a single provider for your keys. By not tying yourself to a single key management solution, you help improve your applications’ availability. This approach also might help if you have compliance, data loss prevention, or disaster recovery requirements that require multiple providers.

You can use the same technique demonstrated previously in this post to encrypt your data to an escrow or additional decryption master key that is independent of your primary provider. This example demonstrates how to use an additional master key, which is an RSA public key randomly generated upon request. (Storing and managing the RSA key pair are out of scope for this blog.)

Encrypt the data with a public master key

Just like the previous code example that created a number of KMS master keys to encrypt data, the following code example creates one more master key for use with the RSA public key.

import aws_encryption_sdk
from aws_encryption_sdk.internal.crypto import WrappingKey
from aws_encryption_sdk.key_providers.raw import RawMasterKeyProvider
from aws_encryption_sdk.identifiers import WrappingAlgorithm, EncryptionKeyType
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa


class StaticRandomMasterKeyProvider(RawMasterKeyProvider):
    """Randomly generates and provides 4096-bit RSA keys consistently per unique key id."""
    provider_id = 'static-random'

    def __init__(self, **kwargs):
        self._static_keys = {}

    def _get_raw_key(self, key_id):
        """Retrieves a static, randomly generated RSA key for the specified key id.

        :param str key_id: Key ID
        :returns: Wrapping key which contains the specified static key
        :rtype: :class:`aws_encryption_sdk.internal.crypto.WrappingKey`
        """
        try:
            static_key = self._static_keys[key_id]
        except KeyError:
            private_key = rsa.generate_private_key(
                public_exponent=65537,
                key_size=4096,
                backend=default_backend()
            )
            static_key = private_key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.PKCS8,
                encryption_algorithm=serialization.NoEncryption()
            )
            self._static_keys[key_id] = static_key
        return WrappingKey(
            wrapping_algorithm=WrappingAlgorithm.RSA_OAEP_SHA1_MGF1,
            wrapping_key=static_key,
            wrapping_key_type=EncryptionKeyType.PRIVATE
        )


def get_multi_master_key_provider():
    # Create multiregion KMS master key provider
    multi_master_key_provider = build_multiregion_kms_master_key_provider()

    # Create static master key provider and add a key
    static_key_id = os.urandom(8)
    static_master_key_provider = StaticRandomMasterKeyProvider()
    static_master_key_provider.add_master_key(static_key_id)

    # Add static master key provider to KMS master key provider
    multi_master_key_provider.add_master_key_provider(static_master_key_provider)

    return multi_master_key_provider, static_master_key_provider

Decrypt the data with the private key

The following decryption code example uses the static RSA master key provider generated previously to demonstrate decryption with a non-AWS master key.

def cycle_data(input_data):
    # Create multi-source master key provider
    multi_master_key_provider, static_master_key_provider = get_multi_master_key_provider()

    # Encrypt data with multi-source master key provider
    ciphertext, header = aws_encryption_sdk.encrypt(
        source=input_data,
        key_provider=multi_master_key_provider
    )

    # Decrypt data using only static master key provider
    plaintext, header = aws_encryption_sdk.decrypt(
        source=ciphertext,
        key_provider=static_master_key_provider
    )

Conclusion

Envelope encryption is powerful, but traditionally, it has been challenging to implement. The new AWS Encryption SDK helps manage data keys for you, and it simplifies the process of encrypting data under multiple master keys. As a result, this new SDK allows you to focus on the code that drives your business forward. It also provides a framework you can easily extend to ensure that you have a cryptographic library that is configured to match and enforce your standards.

We are excited about releasing the AWS Encryption SDK and cannot wait to hear what you do with it. If you have comments about the new SDK or anything in this blog post, submit a comment in the “Comments” section below. If you have questions about the the AWS Encryption SDK, file an issue in the GitHub repos for Java or Python, or read and post on the AWS Crypto Tools Discussion Forum.

– Matt

Want more AWS Security news? Follow us on Twitter.