How to audit an Amazon S3 bucket’s default encryption configuration at scale

Encrypting data at rest fulfills compliance and security standards while providing an extra layer of defense to protect against unauthorized access. As organizations scale, it is critical to develop standardize encryption to minimize the administrative burden of managing encryption keys. Organizations that lack encryption standardization may find themselves unable to access critical data when required. Many customers who are in the process of implementing encryption standardization often ask how they can determine which resources are not conforming with their standardization practices.

Amazon S3 allows you to configure default server-side encryption (SSE) at the bucket-level to automatically encrypt objects when PUT, POST, and COPY requests are made. There are multiple encryption configurations available with Amazon S3. You can configure default encryption to use either an S3-managed key (SSE-S3) or AWS Key Management Services keys (SSE-KMS). When using SSE-KMS, you can use either an AWS managed key or a customer managed key. With SSE-KMS keys, Amazon S3 uses a unique data key for each object. Therefore, Amazon S3 makes a request to AWS KMS each time an encrypted object is requested. This can lead to an increase in AWS KMS costs. Amazon S3 bucket keys is a cost savings feature that you can enable on buckets that are using SSE-KMS for default encryption

In this post, I describe how to audit an Amazon S3 bucket’s default encryption configuration at scale with a Boto3 script. Automating the audit of your encryption keys to run at scale helps reduce the time and effort required to audit large number of resources and makes it easier to track changes over time which is critical to ensure the security of your resources. The provided script is designed to retrieve the default encryption configuration, KMS key ARN, KMS key type (AWS managed key or customer managed key), and the Amazon S3 Bucket Key configuration on all buckets across all Regions. Finally, I will show you how to analyze the output to correlate the SSE configurations for each bucket. This will allow you to ensure all resources are conforming to encryption standardization while also allowing you to identify any rouge encryption keys that are in use in an automated and scalable manner.

Solution overview

AWS Config and S3 Storage Lens can be useful tools to determine if buckets are configured to use SSE-KMS or if buckets are unencrypted. However, when attempting to perform a deeper analysis of your bucket’s encryption configurations, you may find that the available options do not scale well. For example, AWS Config and S3 Storage Lens are unable to determine if buckets, in all AWS Regions, are configured to use:

• SSE-S3 or SSE-KMS
• An SSE-KMS AWS managed key or customer managed key
• Amazon S3 Bucket Key

The provided Boto3 script will perform the ListBuckets, GetBucketLocation, GetBucketEncryption API calls to all buckets across all regions. The script will then issue a DescribeKey API call to the AWS KMS key. The results of these API calls will then be outputted into an organized table.

The following services are used to audit the default server-side encryption mode:

• Amazon S3
• AWS KMS

This script will perform read-only calls. No changes will be made to your resources.

Prerequisites

You should have the following prerequisites:

• An AWS account.
• Amazon S3 bucket.
• Python3 installed on your local machine.
• AWS credentials to access your AWS account.
• Permissions to perform the following actions:
  • s3:ListAllMyBuckets
  • s3:GetBucketLocation
  • s3:GetEncryptionConfiguration
  • kms:DescribeKey

The IAM policy for the IAM user or role that is running this script needs to have the following minimum Amazon S3 and AWS KMS permissions to retrieve the required information.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3BucketListingBucketLocationsBucketEncryptConfig",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetEncryptionConfiguration",
        "s3:ListAllMyBuckets"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Sid": "AllowAccesstoKmsKeyMetadata",
      "Action": [
        "kms:DescribeKey"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:kms:*"
    }
  ]
}

You also need to ensure that the SSE-KMS customer managed keys allow the IAM role or user access as well. For more information about creating a KMS key policy, visit the documentation.

If you have Amazon S3 buckets in opt-in AWS Regions, the IAM role or user must have credentials that are enabled for these AWS Regions.

Walkthrough

• Click here to download the script.
• Open a terminal session on the device where the script is saved.
• Execute the script by running the following command:

$ python3 audit_s3_default_encryption.py

• When prompted, input the output location for the report.
• Wait for the script to complete.
• Navigate to the output location and open the report.

Analyze the report

The following output is an example of the type of results generated by the script:

Column B can contain the following values:

• AWS KMS Key ARN: provides the ARN for the KMS Key that is configured in the bucket’s default encryption configuration.
• AWS KMS Key Alias: provides the alias for the KMS Key that is configured in the bucket’s default encryption configuration.
• AES256: indicates that the bucket is configured with SSE-S3 encryption.
• SSEConfigNotFound: indicates that the bucket has no default encryption configurations.
• AccessDenied: indicates that the IAM user or role does not have the required permissions to perform the GetBucketEncryption API call.

Column C can the contain the following values:

• CUSTOMER: indicates that the AWS KMS Key is an SSE-KMS customer managed key.
• AWS: indicates that the AWS KMS Key is an SSE-KMS AWS managed key.
• N/A: indicates that SSE-KMS is not configured.
• AccessDenied: indicates that the IAM user or role does not have the required permissions to perform the DescribeKey API call or that the AWS KMS key is located in a different AWS Region than the Amazon S3 bucket.
• Unknown: indicates that the DescribeKey API call could not be performed because the GetBucketEncryption API call failed.
• KeyNotFound: indicates the DescribeKey API call failed due an invalid AWS KMS key.

Column D can the following values:

• Empty: indicates that SSE-KMS is not configured on this bucket; Bucket Key is not being used.
• True: indicates that Bucket Key is configured on this bucket.
• False: indicates that Bucket Key was manually set to false.
• AccessDenied: indicates that the GetBucketEncryption API call failed.

This report can be used to standardize your default bucket encryption across all AWS Regions. You will be able to easily identify buckets that do not conform with your standardization requirements and identify buckets that do not have Amazon S3 Bucket Key enabled. Using the findings from this report, you will be able to create a plan to remediate buckets that do not follow your standardization requirements and buckets that are not taking advantage of Amazon S3 Bucket Key.

Conclusion

In this post, I showed how you can audit your Amazon S3 bucket’s default encryption configuration at scale by using a Boto3 script. This can help you understand if you need to implement encryption standardizations or if encryption standardization practices are being followed. Using the generated output, you will be able to correlate all the different encryption types on all buckets across all regions. Additionally, you will be able to understand which buckets are configured with Amazon S3 Bucket Key to ensure cost optimization features are enabled. Finally, you can also use this script to track which buckets have SSE-S3 enabled and which ones are still pending.

It is important to standardize encryption across your organization to simplify management and security. Thus, when using Amazon S3, take steps to secure you data by auditing your encryption configuration with the provided script.

Thanks for reading this blog, if you have any comment or questions, don’t hesitate to leave them in the comments section.