How to enable multi-account log collection with AWS Control Tower and Cribl

As organizations grow and expand their Amazon Web Services (AWS) footprint, managing logs across multiple accounts can become complex. With Cribl, you can use a streamlined solution to simplify this process for customers using AWS Control Tower. In this post, I walk you through how to subscribe to Crible Cloud in AWS Marketplace and how to enable log collection from multiple AWS accounts within an AWS Organizations account using Cribl and AWS Control Tower.

In the previous post, Automate multi-account observability in AWS using Cribl and AWS Control Tower, users were encouraged to deploy local worker groups in each AWS account. This pattern can be used if there is no way to establish a trust with those local accounts. The approach we describe in this post removes the need for worker groups in each AWS account and instead uses the centralized logging account in the landing zone deployed in AWS Control Tower to pull the log events from a centralized Amazon Simple Storage Service (Amazon S3) bucket.

Prerequisites

You must complete the following prerequisites before implementing the Cribl and AWS Control Tower integration solution:

1. Subscribe to Cribl Cloud in AWS Marketplace. AWS Marketplace is a digital catalog that makes it easy to find, buy, deploy, and manage third-party software you need to build solutions and run your business on AWS. The platform facilitates billing, invoicing, and payments, making it seamless for customers to handle all transactions directly through their AWS account.
2. In the upper right, choose View purchase options and select your option to subscribe.

Solution overview

The updated AWS Control Tower integration with Cribl is based on the centralized logging account within an AWS Control Tower deployment. The integration consists of one AWS CloudFormation template that automates the provisioning, setup, and integration of all the components necessary for this solution.

The AWS CloudFormation template and the detailed README for this solution are available in the Cribl Cloud Trust IAM Role CloudFormation Template repository. The template is deployed in the AWS Control Tower logging account and performs the following actions:

1. Creates an AWS Identity and Access Management (IAM) role named CriblTrustCloud
2. Configures a trust relationship in this IAM role with Cribl.Cloud’s AWS account
3. Attaches an IAM policy that grants access to the centralized AWS Control Tower S3 bucket that holds all the logging data for the customer’s Organizations account and Amazon Simple Queue Service (Amazon SQS) resources
4. Creates an Amazon S3 event notification in the S3 bucket that holds the logging data sending a notification to the SQS queue created and to your Cribl account
5. Outputs the role name, Amazon Resource Name (ARN), and an external ID for authentication

The following diagram illustrates the solution architecture.

Figure 1: Cribl multi-account collection

Solution walkthrough: Enabling multi-account log collection with AWS Control Tower and Cribl

The first step is to create a trust relationship between your centralized AWS Logging Account and the Cribl.Cloud tenant. To deploy the solution, follow these steps:

1. In the cribl-aws-cloudformation-templates directory, locate the CloudFormation template for CloudTrail log data. For VPC Flow Logs, you can use this CloudFormation template.
2. Use the templates to deploy a CloudFormation stack in your AWS logging account.

If you use AWS Key Management Service (AWS KMS) encryption for your CloudTrail data, you need to add KMS key policy permission to allow the Cribl IAM role to decrypt the log file objects to send to your Cribl Cloud account. Follow these steps:

1. To check you KMS key details, on the CloudTrail console on the Trails tab, choose your centralized CloudTrail trail. Navigate to it from the General details For VPC Flow Logs, on your centralized S3 bucket for your VPC Flow Logs, check your encryption key ARN under Default encryption.
2. In the templates/s3bucketcollection/template/ directory, download the json file and replace the following placeholders:
   1. [CriblTrustCloud.Arn] with the ARN output from the CloudFormation template you executed in the previous step
   2. [Logging-Account-ID] with the AWS account ID of your logging account
   3. [CloudTrail-S3-Bucket-Name] with your centralized CloudTrail S3 bucket name
3. Add the modified KMS key policy JSON to your KMS key policy for your CloudTrail. Alternatively, deploy the KMS Key Policy Update CloudFormation template downloaded from the templates/s3bucketcollection/template/ directory to the AWS account that manages your KMS key used in your centralized CloudTrail trail.

After the trust relationship is established, the next step is to connect the S3 bucket containing your organization’s logging data to Cribl. This lets Cribl ingest, process, and route your logs from a central location. Follow these steps:

1. Log in to your Cribl Cloud instance.
2. In the Sources section, add a new S3 source.
3. Configure the source to point to your centralized logging bucket by adding the SQS queue created in the CloudFormation template, as shown in the following screenshot.

Figure 2: The General Settings page in the Cribl.Cloud instance

4. In the navigation pane, choose AssumeRole and select Enable for Amazon S3 and Enable for Amazon SQS, as shown in the following screenshot. Add your AWS account ID and the IAM role ARN and external ID created in the CloudFormation template. You can find this one the CloudFormation console on the Output

Figure 3: The AssumeRole page

Test your integration

After the S3 bucket has been added to your Cribl UI S3 source, choose the new source and then choose Status. Both workers should show green checkmarks. If they are red Xs, expand each worker to review the error code. Typical issues can include not having the proper role added to the setup or forgetting the external ID.

To check that the integration is working, perform the following steps:

1. Sign in to the AWS Control Tower managed account. Navigate to the CloudFormation console.
2. Check that there is an AWS CloudFormation stack in your centralized AWS logging account that has successfully deployed the Cribl.Cloud trust resources.
3. In the navigation pane, select this stack instance and choose Stack info. The status field should display a value of CREATE_COMPLETE.

Clean Up

To clean up the solution go to the AWS CloudFormation Console, select the CloudFormation stack you used to create the Cribl trust IAM Role and select Delete Stack.

Benefits of this approach

By implementing this solution, you’ll gain the following benefits:

• Centralized visibility – Collect and analyze logs from all your AWS accounts in one place.
• Automated deployment – New accounts are automatically configured for log collection.
• Cost optimization – Route only the most valuable data to your analysis tools, reducing storage and processing costs.
• Compliance – Apply your logging and monitoring standards to all accounts.

Conclusion

Managing logs across a multi-account AWS environment doesn’t have to be difficult. With Cribl’s integration with AWS Control Tower, you can automate the deployment of a comprehensive, scalable logging infrastructure. This approach simplifies management and provides flexibility to route, shape, and enrich your data before it reaches its destination.

Remember, the key to effective observability in a complex cloud environment is not only collecting all the data but collecting the right data and making it actionable. Cribl provides the control and flexibility you need to make the most of your telemetry data.

For more detailed information on setting up and configuring this solution, review the Cribl documentation or reach out to the support team. To take control of your telemetry data and optimize your AWS infrastructure, visit the Cribl solution portfolio available in AWS Marketplace.

About the Authors

Gabriel Costa

Gabriel Costa is a Senior partner solutions architect at AWS, working with AWS Partners and customers on all things cloud operations. Outside of work, he enjoys playing the guitar, reading about philosophy, watching sci-fi and anime, and searching with his wife for the new cool restaurant in town.

Kamilo “Kam” Amir

Kamilo “Kam” Amir is the Director of Business Development for Cribl and is based in the Washington, D.C. area. He’s been with Cribl since LogStream version 2.2 and leads the technical alliances program. If you need to find him, just look for him hiking in Rock Creek Park with his family and husky or in the Cribl Slack Community.