Aggregating logs with S3 Same-Region Replication

Aggregating logs to a secure dedicated location streamlines critical operations like Security Information and Event Management (SIEM). As more customers adopt a multi-account strategy, central logging becomes a key component of driving operational excellence. Considering the benefits, enterprise customers often use complex third-party tools to build their centralized log aggregation solution. With the announcement of Amazon S3 Same-Replication (SRR), customers can now use this feature for a multitude of use cases including log aggregation. S3 SRR is a feature of S3 Replication that automatically replicates data between buckets within the same AWS Region. You can configure SRR to replicate new objects uploaded to a specific bucket or prefix. SRR can also replicate new objects with specific tags to a destination of your choice. Customers can also use SRR to aggregate logs from different S3 buckets, owned by a single account or multiple accounts, into one centralized bucket for further processing.

In this blog post, I demonstrate log aggregation by collecting VPC Flow Logs from VPCs hosted in a multi-account landscape using S3 SRR. This setup uses two AWS accounts, Account A and Account B for the purposes of this tutorial, but the same principles apply for data aggregation in a single account scenario. In this setup, the account generating the VPC Flow Logs is referred to as Account A and the account hosting the aggregated log bucket is referred to as Account B.

Prerequisites

Let’s review the prerequisites before diving deeper into the step-by-step guide of setting up S3 Same-Region Replication.

Account A prerequisites

1. VPC Flow Logs can be set to publish logs to pre-created S3 buckets. You can either use an existing S3 bucket or create a dedicated bucket following the steps from the How do I create S3 buckets page. I am using the S3 bucket named: “source-bucket-for-replication” for this example.
2. Enable versioning on this S3 bucket following the steps from How Do I Enable or Suspend Versioning for an S3 Bucket page.
3. Once the bucket is ready, follow the steps from the Publishing Flow Logs to S3 page for enabling VPC Flow Logs. Choose source-bucket-for-replication as the flow log destination.

1. • Note: Enabling VPC Flow Logs adds a bucket policy to allow delivery.logs.amazon.com to publish logs to the above bucket.

Account B prerequisites

1. Create an S3 bucket in Account B. This S3 bucket is the central bucket holding the VPC Flow Logs from Account A as well as the VPC Flow Logs from Account B. I am using the S3 bucket named “destination-bucket-for-replication” for this example.
2. Enable versioning on this S3 bucket following the above steps.
3. Enable VPC Flow Logs in Account B following the above steps. Choose destination-bucket-for-replication as the flow log destination.

S3 Same-Region Replication setup

Follow these steps for setting up S3 SRR:

In Account A

1. From the AWS Management Console, navigate to the “source-bucket-for-replication” bucket and verify the bucket policy from the Permissions tab. The attached bucket policy is a result of pointing this bucket as VPC Flow Logs destination.

2. Navigate to the Management tab and select the Replication option. Choose Add rule for adding a replication rule.

3. In the Replication Rule dialog box, select the Entire bucket option and choose Next. Selecting the Entire bucket option replicates all the objects uploaded to the bucket.

Note: You can replicate objects with the same prefix or same tags, using the Prefix or tags options.

4. Under the destination bucket, drop down the Select Bucket option and select the Buckets in another account option. Provide the AWS Account ID and the name of the destination bucket and choose Save.

5. Select the appropriate Destination options for bucket objects and choose Next.

6. For the IAM role, you can select an existing role or create an IAM role that provides sufficient permissions to S3 for replicating objects from the “source-bucket-for-replication” bucket to the “destination-bucket-for-replication” bucket. For this example, choose Create new role and provide the Rule name.

7. Copy the generated bucket policy by clicking Copy. This bucket policy should be added to the “destination-bucket-for-replication” bucket in Account B. Choose Next. Review the provided details on the next page and click Save.

In Account B

1. Navigate to the S3 console and select the destination-bucket-for-replication bucket.
2. Navigate to the Permissions tab and choose Bucket Policy. Paste the bucket policy you got from Step-5 of the Account A setup and choose Save.

3. Under the Management tab, choose Replication and drop down the Actions menu. Select Receive Objects… from the menu as shown below.

4. On the Receive Objects dialog box, provide the account ID of the source account and choose Done.

5. This action updates the bucket policy on the “destination-bucket-for-replication” bucket, providing access to the root of the source bucket account to replicate objects to this bucket.
6. Navigate to the bucket Overview tab and observe the logs from the source account show up in this bucket. Logs from both accounts will be in the “AWSLogs” folder.

AWS CLI and SDK support

Follow this step-by-step tutorial for setting up S3 replication using AWS CLI and SDK.

Cleanup

Follow these steps if you must delete the VPC Flow Logs and SRR configurations:

• In both Account A and Account B, delete the enabled VPC Flow Logs. Here is an example of deleting VPC Flow Logs from Account A. Follow the same steps in Account B as well:

• In Account A, for deleting the SRR rule, navigate to the Management tab of the “source-bucket-for-replication” bucket. Choose Replication, select the rule-to-replicate-VPC-Flow-Logs replication rule and delete the rule.

• The above steps delete the rules but still keeps the data intact. So, navigate to the Overview tab of each bucket and make sure to clean up the data for avoiding any unwanted charges.

Conclusion

In this post, we reviewed how S3 SRR can be used for log aggregation. With S3 SRR enabling customers to make identical copies of S3 objects uploaded to the S3 bucket within the same AWS Region, this feature can be extended to many other use cases including in-region processing of replicated data, or configuring live replication between test and development environments. SRR can also help address data sovereignty and compliance requirements by keeping a copy of your objects in the same AWS Region as the original. For more information, see S3 Replication documentation and share your experience.