Modern data protection architecture on Amazon S3: Part 1

Update (12/11/2023): As of November 20, 2023, Amazon S3 supports enabling S3 Object Lock on existing buckets.

Keeping data secure and usable in unforeseen circumstances like accidental breaches, human error, and hacking is critical to business continuity and success. To effectively mitigate the impact of these events on business-critical assets, one of the recommended strategies is creating immutable, unchangeable copies of those assets and storing them in isolated, secondary accounts with restricted access.

In this two-part blog series, we first walk through creating immutable copies of your data and replicating those immutable copies from one Amazon S3 bucket to another. In part 2, we walk through how you can selectively retrieve the copies if the primary data source has been compromised.

In this first post, we walk through implementing a solution to protect your business-critical assets stored in Amazon S3 by using Amazon S3 Replication and S3 Object Lock to create an immutable copy of your data in a separate AWS account. This helps you have an isolated redundant copy of critical data outside of your primary account using just native features of S3. Besides making the solution easier to implement, the cost can be optimized through the use of features like the S3 Intelligent-Tiering storage class as it delivers automatic storage cost savings for unknown or changing data access patterns without performance impact or operational overhead and no retrieval fees (except for Archive Access tier expedited requests). Customers would see this approach of having segregated primary and secondary accounts help to limit the impact of data breaches and data corruption while keeping the retrieval mechanism simplistic.

Solution overview

Our objective across this two-part series is to present an architectural solution to securely store and retrieve immutable copies of data using native Amazon S3 features like Amazon S3 Same-Region Replication (SRR) and Amazon S3 Object Lock. With the usage of these S3 features, we can provide an added layer of protection to business critical assets.

Along with the native Amazon S3 features, this solution leverages certain key additional services for data security and incident alerting.

Using Amazon S3 Same-Region Replication

In this solution, we use S3 SRR to create object copies in another account. If you currently have data stored in your S3 bucket and must replicate existing objects, you can use S3 Batch Replication. We recommend that you consider what specific data requires this level of protection because you will now be storing duplicate copies leading to an increase in the overall cost of your storage. Many organizations will choose to replicate only their most business-critical data.

Using Amazon S3 Object Lock

For immutable data storage, we use Amazon S3 Object Lock. S3 Object Lock uses a write-once-read-many (WORM) model for storing objects in S3. When enabled, it can help prevent objects from being deleted or overwritten. There are two modes to consider for data protection when using Object Lock. Governance mode allows you to alter the lock settings or delete an object when granted special permissions. On the other hand, compliance mode locks the object without the ability for anyone, including the root user of your AWS account, to alter the lock or retention period.

Note that changing a retention period on an object with S3 Object Lock enabled or enabling S3 Object Lock on existing objects will incur a PUT API operation and associated costs.

Also, deleting an object without a version id specified will keep the delete marker of that object as the current version instead of completely removing the object. However, the S3 GET API will return an 404 error on trying to access the current version stating that the object cannot be found. To get further insights into how to use S3 SRR with Object lock at the bucket level versus the object level, refer to the S3 object lock management documentation.

You can enable S3 Object Lock at the time of bucket creation or you can contact AWS Support if enabling Object Lock on an existing bucket. Also, note that the replicated objects (in Compliance mode) will retain data in the destination bucket that can’t be deleted. This would incur additional storage costs, so you should choose a storage class and retention period that balances the need for additional data copies with the cost of retaining them.

How versioning and delete marker works

Using AWS Key Management Service

Along with enabling S3 Object Lock, we also need to consider securing those objects at rest. For securing data at rest, we use Amazon S3 Server-Side Encryption (SSE) with AWS Key Management Service (KMS) – (SSE-KMS) – and Amazon S3 Bucket Keys. S3 Bucket Keys can reduce the request costs SSE-KMS by up to 99% by decreasing the request traffic from S3 to KMS.

The reason why we recommend SSE-KMS over SSE-S3 is the additional level of security provided by the permissions on the KMS key itself that enables decryption by only authorized users and applications.

Your source bucket might contain objects that were created with server-side encryption by using SSE-KMS. By default, Amazon S3 doesn’t replicate these objects. You can optionally direct Amazon S3 to replicate these objects by specifying the AWS KMS key (for the AWS Region of the destination bucket) to use for encrypting object replicas. Please refer to the AWS documentation for more details.

Using Amazon EventBridge

After securing the buckets through the recommended approach, we now need to set up alert notifications to capture real time incident alerts. This will enable us to initiate the remediation actions to stop replication if need be. In order to set up the notifications for S3 API operations, we use Amazon EventBridge, which is a serverless event bus that makes it easier to build event-driven applications at scale using events generated from your applications, integrated software as a service (SaaS) applications, and AWS services. If an event takes place, we want to setup an automated response to stop the replication. We have CloudWatch alarm to monitor abnormally elevated PutRequests API metrics. Once, the alarm is triggered, the Amazon EventBridge rule captures the alert and triggers an AWS Lambda function to terminate the replication.

In this solution, we recommend using EventBridge over other options because EventBridge can provide the capabilities of content-based filtering as well as input transformation while seamlessly integrating with the other services used in this solution.

So, in the event of a security incident like a ransomware attack at the source, we can stop the replication of compromised files to the target buckets at the earliest possible time. We will go over the details on this in the section of this blog titled “Automated disabling of S3 Replication rule.”

The architectural layout replicating the assets into isolated destination account

Solution walkthrough

Let’s take a quick look into the following sections and their functional requirements to build up the solution:

1. AWS Key Management Service(KMS) Key creation: Highlights the creation and use of encryption key to store objects in encrypted format.
2. Bucket creation: Shows how we create buckets at the source and the target AWS accounts and restrict the access.
3. S3 Replication: Details the steps to set up Same-Region Replication from the source account to the destination account.
4. Update AWS KMS and Identity and Access Management(IAM) permissions: Details on how to grant permissions for successful replication setup.
5. Automated disabling of S3 Replication rule: Provides the steps on how to disable replication automatically when a ransomware attack has been detected.

Here we provide step-by-step guidance on how to create a secret key so that the objects created in your bucket are encrypted by default and can only be restored from the target (destination) account if the correct encryption keys are used and access is granted to those keys. In this solution, the source objects are encrypted with source bucket S3 Bucket Keys, and the object replicas are encrypted with the destination bucket S3 Bucket Keys.

1. AWS Key Management Service (KMS) key creation

In this section, you create the AWS KMS keys in the source and destination accounts used to encrypt objects and object replicas.

1. In the source account, create a new customer managed AWS KMS key *(Symmetric) in the same Region you plan to create your source bucket.
2. In the destination account, create a new customer-managed AWS KMS key (Symmetric) in the same Region you plan to create your destination bucket.
3. Allow the source account AWS KMS to use your destination bucket AWS KMS key

In the destination account AWS KMS console, add the source AWS account number in the Other AWS Accounts section of the AWS KMS key you created in step 2.

*When you create a AWS KMS key, by default, you get a KMS key for symmetric encryption. This is the basic and most commonly used type of KMS key.In AWS KMS, a symmetric encryption KMS key represents a 256-bit encryption key that never leaves AWS KMS unencrypted. 

2. Bucket creation

In this section, you create a source and destination bucket with server-side encryption using AWS KMS keys (SSE-KMS) and S3 Bucket Keys for objects stored in Amazon S3.

1. In the source account, create a new S3 bucket. This is your main bucket for storing data you are looking to protect against ransomware incidents.

a. Select ACLs disabled (recommended) for Object Ownership. Details on disabling ACLs can be found in the Amazon S3 User Guide.
b. Select Block all Public access.
c. Select Enable for Bucket Versioning.
d. Enable Object Lock.
e. Select Enable for Default Encryption.

i. Choose AWS Key Management Service key (SSE-KMS) for Encryption key type.
ii. For AWS KMS key, select Choose from your AWS KMS keys and select the KMS key you created earlier.
iii. Select Enable for Bucket Key.

2. In the destination account, create another S3 bucket. This is your bucket that will store immutable copies of the objects from the source account.

a. Select ACLs disabled (recommended) for Object Ownership.
b. Select Block all Public access.
c. Select Enable for Bucket Versioning.
d. Enable Object Lock.
e. Select Enable for Default Encryption.

i. Choose AWS Key Management Service key (SSE-KMS) for Encryption key type.
ii. For AWS KMS key, select Choose from your AWS KMS keys and select the KMS key you created earlier.
iii. Select Enable for Bucket Key.

3. After the destination bucket creation is complete, go to the Properties of the destination bucket, and under Object Lock, enable the Default retention period. Configure Default retention mode and Default retention period for Object Lock with the following settings:

a. Default Retention = Enable
b. Default retention mode = Governance

• • We use Governance mode in this example so you can remove the lock if needed. For production buckets, we recommend Compliance mode for ransomware protection.

c. Default retention period = 7 days

• • Recommendation is to use 7/15/30 days, but here we use 7 days for the purpose of demonstration and to prevent locking the file for an extended period of time and incurring additional storage costs

3. S3 Replication

In this section, configure S3 Same-Region Replication to replicate objects from the source to the destination account.

1. In the source account S3 console, select the main bucket you created earlier and select the Management In the Replication rules section, select Create replication rule.
2. Enter a Replication rule name and the following options:

a. Status = Enabled
b. In the Source bucket section, select Apply to all objects in the bucket.

• • In a production environment, we recommend setting a filter to limit what is in scope for replication.

c. For Destination, choose Specify a bucket in another account and enter the destination account number and the protected bucket name.
d. Enable Change object ownership to destination bucket owner.
e. For IAM role, select Choose from existing IAM roles and select Create new role
f. In Encryption, check the box for Replicate object encrypted with AWS KMS.
g. Select Enter AWS KMS key ARN and enter the destination account KMS Amazon Resource Name (ARN) for the KMS key you created earlier.
h. Choose Change the storage class for the replicated objects and select S3 Intelligent-Tiering for the destination. Refer to the Amazon S3 documentation for more details on storage classes.

3. In Additional replication options, enable Replication metrics and notifications to monitor your replication with Amazon CloudWatch. (S3 Replication metrics are billed at the same rate as Amazon CloudWatch custom metrics.) Replication metrics include Bytes Pending Replication, Replication Latency, and Operations Pending Replication. By enabling these metrics, you can monitor the minute-by-minute progress of your replication and use them to help with troubleshooting.

a. There are additional options here: Replication Time Control (RTC), Delete marker replication, and Replica modification sync. As we are implementing replication for ransomware protection, you would not want to enable Delete marker replication as this could potentially replicate unwanted deletes from the source bucket and create delete markers in the destination. This will make the delete marker the current version in the replica, making it appear that the objects have been deleted and could require the removal of the replicated delete markers for simplified restoration. Replica modification sync replicates metadata changes made to the replica copies back to the source object, making the replication bidirectional. Since the destination contains golden copies, there should not be any changes made to those objects to replicate back to the source. Replication Time Control (RTC) replicates most objects that you upload to Amazon S3 in seconds and is designed to replicate 99.99 percent of those objects within 15 minutes. RTC helps you meet compliance or business requirements for data replication and, by default, includes S3 Replication metrics and S3 Event Notifications. There is an additional cost per GB for data replicated with S3 RTC.

4. Select Save. You may be prompted with an option to Replicate existing objects? If you have existing objects, you can choose this option to replicate with S3 Batch Replication. For this example, choose No, do not replicate existing objects, and select Submit. Your replication rule should look like the following screenshot.

4. Update AWS KMS and Identity and Access Management (IAM) permissions

In this section, you update the permissions needed for replication to complete successfully. When using S3 Replication, AWS KMS, and S3 Bucket Keys together, you need to update permissions in AWS KMS, IAM, and Amazon S3.

1. In the source account, open the AWS KMS console and locate the key you created earlier. Add the new s3crr* role created by the replication rule as a Key user. The newly created role should be named s3crr_role_for_<source bucket name>.

*s3crr_role_for prefix is attached to the replication role.

2. Open the Identity and Access Management (IAM) console and locate the role created by the replication rule. The newly created role should be named s3crr_role_for_<source bucket name>.
3. You will need to update the IAM policy to include AWS KMS key permissions and update resources for use with S3 Bucket Keys. You can copy the following example and update it accordingly for your bucket and AWS KMS key names and ARNs.

a. Since we use bucket keys for AWS KMS in this example, the kms:EncryptionContext:aws:s3:arn in the following IAM policy must use the bucket ARN and not the object ARN (/*).
b. Items in bold need to be changed with your values

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Action":[
            "s3:ListBucket",
            "s3:GetReplicationConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging",
            "s3:GetObjectRetention",
            "s3:GetObjectLegalHold"
         ],
         "Effect":"Allow",
         "Resource":[
            "arn:aws:s3:::<source bucket>",
            "arn:aws:s3:::<source bucket>/*",
            "arn:aws:s3:::<destination bucket>",
            "arn:aws:s3:::<destination bucket>/*"

         ]
      },
      {
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags",
            "s3:GetObjectVersionTagging",
            "s3:ObjectOwnerOverrideToBucketOwner"
         ],
         "Effect":"Allow",
         "Condition":{
            "StringLikeIfExists":{
               "s3:x-amz-server-side-encryption":[
                  "aws:kms",
                  "AES256"
               ]
            }
         },
         "Resource":[
            " arn:aws:s3:::<destination bucket>/*"
         ]
      },
      {
         "Action":[
            "kms:Decrypt"
         ],
         "Effect":"Allow",
         "Condition":{
            "StringLike":{
               "kms:ViaService":"s3.<source bucket region>.amazonaws.com",
               "kms:EncryptionContext:aws:s3:arn":[
                  "arn:aws:s3:::<source bucket>"
               ]
            }
         },
         "Resource":[
            "<source bucket KMS Key ARN>"
         ]
      },
      {
         "Action":[
            "kms:Encrypt"
         ],
         "Effect":"Allow",
         "Condition":{
            "StringLike":{
               "kms:ViaService":[
                  "s3.<destination bucket region>.amazonaws.com"
               ],
               "kms:EncryptionContext:aws:s3:arn":[
                  "arn:aws:s3:::<destination bucket>"
               ]
            }
         },
         "Resource":[
            "<destination bucket KMS key ARN>"
         ]
      }
   ]
}

4. In the destination account, create a bucket policy to allow replication from the source account bucket. Items in bold need to be changed for your values

{
   "Version":"2012-10-17",
   "Id":"",
   "Statement":[
      {
         "Sid":"Set permissions for objects",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<AWS Account>:role/service-role/s3crr_role_for_<source bucket name>"
         },
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete"
         ],
         "Resource":"arn:aws:s3:::<destination bucket>/*"
      },
      {
         "Sid":"Set permissions on bucket",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<AWS Account>:role/service-role/s3crr_role_for_<source bucket name>"
         },
         "Action":[
            "s3:List*",
            "s3:GetBucketVersioning",
            "s3:PutBucketVersioning"
         ],
         "Resource":"arn:aws:s3:::<destination bucket>"
      },
      {
         "Sid":"Set owner on objects in bucket",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<AWS Account>:role/service-role/s3crr_role_for_<source bucket name>"
         },
         "Action":"s3:ObjectOwnerOverrideToBucketOwner",
         "Resource":"arn:aws:s3:::<destination bucket>/*"
      }
   ]
}

5. Test by uploading objects to the source bucket and verify that the object successfully replicates to the protected bucket, is in the configured storage class, and the object is locked with S3 Object Lock.

5. Automated disabling of S3 Replication rule

This section shows how to create a CloudWatch alarm to monitor PUT events in the source bucket. Elevated PUT request rate may point to security incidents like a potential Ransomware activity in your environment (we use value of 10 PUT requests per minute in this blog as an example). If the alarm threshold is exceeded, a Lambda function will stop your S3 Replication rule. You could also create a similar function to disable replication on delete operations. Remember to utilize a threshold value considering the average count of your PUT requests. This automation can prevent unwanted data from replicating to your destination bucket during a security incident.

Architecture diagram illustrating disabling of replication

1. In the CloudWatch console, create a new alarm for S3 PUT (PutRequests) metrics exceed 10 in a 1-minute Period, with a Statistic of SUM.

a. The elevated PutRequests metric values may indicate an anomaly pointing to an active attack.

2. Set up an Amazon SNS email notification.
3. Create a new AWS Lambda function to update the replication rule to a disabled state. With replication rules for Amazon S3, the API can only put a new replication rule. The following function deletes the existing rule and replaces it with a new rule set as disabled.

import json
import boto3

print('Loading function')

s3 = boto3.client('s3')

def lambda_handler(event, context):
    try:
        
        print('event: ' + json.dumps(event))
        
        bucket = '<source bucket name>' # if the lambda is triggered by alarm, the source bucket name will be a part of the event object payload
        
        config = s3.get_bucket_replication(Bucket=bucket)
        print('Bucket {} CRR config: {}'.format(bucket, json.dumps(config['ReplicationConfiguration'])))
        
        new_config = config['ReplicationConfiguration']
        new_config['Rules'][0]['Status'] = 'Disabled'
        
        s3.put_bucket_replication(
            Bucket=bucket,
            ReplicationConfiguration=new_config,
        )
        print('Disabled CRR for bucket {}, config : {}'.format(bucket, json.dumps(new_config)))
        
        return new_config
        
    except Exception as e:
        print(e)
        print('Failed disabling CRR for bucket {}, error {}'.format(bucket, e))
        raise e

4. Create a new Amazon EventBridge rule to call the AWS Lambda function in the event of the Amazon CloudWatch alarm exceeding the threshold.

Cleaning up

If you followed along and want to avoid incurring unexpected charges, remember to delete the source and target buckets, and disable versioning on the source bucket. Also delete the AWS Lambda functions, IAM roles, and AWS KMS keys.

Summary

In this blog post, we demonstrated a solution for protecting your business-critical assets using Amazon S3 Replication and S3 Object Lock to create immutable copies of your data for safe keeping in an AWS account with restricted access. We also covered how Amazon S3 Intelligent-Tiering can allow you to transition your isolated, immutable copies to less expensive storage classes based on access patterns.

The solution demonstrated can enable you to amplify your threat mitigation strategy and protect your business-critical data as it keeps an isolated redundant copy of your critical data outside of primary use accounts. Amazon S3 Intelligent-Tiering enables you to set up this solution while saving on your threat mitigation strategy with automated cost-savings based on access patterns.

In part 2 of this series, we discuss recovering and restoring objects initially replicated using the process outlined here in part 1.

Additional recommendations for ensure the data stored in your source Amazon S3 bucket is safe can be found in the ransomware mitigation strategy whitepaper. If you have any comments or questions, don’t hesitate to leave them in the comments section.