Centralized Logging on AWS will be deprecated on March 1, 2024. It is being superseded by the Centralized Logging with OpenSearch (CLOS) solution. After deprecation, all existing deployments will continue to work but the solution will no longer be supported and maintained. If you’re not using GovCloud region deployments, we highly recommend migrating to version 2.0.0 or newer of the CLOS solution by following the steps outlined in the Centralized Logging on AWS implementation guide.
Overview
The Centralized Logging on AWS solution helps organizations collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. This solution consolidates, manages, and analyzes log files from various sources, such as such as audit logs for access, configuration changes, and billing events. You can also collect Amazon CloudWatch Logs from multiple accounts and AWS Regions.
This solution uses Amazon OpenSearch Service and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, that results in a unified view of all the log events. In combination with other AWS managed services, this solution provides you with a turnkey environment to begin logging and analyzing your AWS environment and applications.
Supported log formats include Amazon VPC Flow Logs, AWS CloudTrail, AWS Lambda, Common Log Format, Space Delimited, JSON, Apache web server logs, and other (user defined) formats.
Benefits
The AWS CloudFormation template automatically launches and configures the components necessary to upload log files from multiple accounts and AWS Regions to Amazon OpenSearch Service for analysis and visualization in a customizable, user-friendly dashboard.
Control access to your dashboards using Amazon Cognito to simplify authentication to Amazon OpenSearch Service.
Extend your logging capabilities beyond default AWS service logs. This flexible solution includes examples for capturing host-level log files and VPC flow logs, and is designed to scale with your growing business.
Simplify data visualization using built-in Amazon OpenSearch Service support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana.
Technical details
The Centralized Logging on AWS solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.
Step 1a - Log ingestion
Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. You can configure CloudWatch Logs subscription filters for log groups to be streamed to the Centralized Logging on AWS account.
Step 1b - Log ingestion
You can deploy an optional demo AWS CloudFormation template to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
Step 2a - Log indexing
A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon OpenSearch Service domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target.
Step 2b - Log indexing
Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon OpenSearch Service document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
Step 3 - Visualization
Amazon OpenSearch Service and Kibana provide data visualization and exploration support. An Amazon OpenSearch Service domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon OpenSearch Service cluster and Kibana dashboard.
Step 1a - Log ingestion
Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. You can configure CloudWatch Logs subscription filters for log groups to be streamed to the Centralized Logging on AWS account.
Step 1b - Log ingestion
You can deploy an optional demo AWS CloudFormation template to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
Step 2a - Log indexing
A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon OpenSearch Service domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target.
Step 2b - Log indexing
Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon OpenSearch Service document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
Step 3 - Visualization
Amazon OpenSearch Service and Kibana provide data visualization and exploration support. An Amazon OpenSearch Service domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon OpenSearch Service cluster and Kibana dashboard.