Centralized Logging

What does this AWS Solutions Implementation do?

The Centralized Logging solution helps organizations collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. This solution consolidates, manages, and analyzes log files from various sources. You can collect Amazon CloudWatch Logs from multiple accounts and AWS Regions.

This solution uses Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, that results in a unified view of all the log events. In combination with other AWS managed services, this solution provides you with a turnkey environment to begin logging and analyzing your AWS environment and applications.

Click here for related content

Benefits

Centralized logging reference implementation

blank

The AWS CloudFormation template automatically launches and configures the components necessary to upload log files from multiple accounts and AWS Regions to Amazon OpenSearch Service for analysis and visualization in a customizable, user-friendly dashboard.

Access to your dashboards using Amazon Cognito

blank

Control access to your dashboards using Amazon Cognito to simplify authentication to Amazon OpenSearch Service.

Logging capabilities beyond default AWS service logs

blank

Extend your logging capabilities beyond default AWS service logs. This flexible solution includes examples for capturing host-level log files and VPC flow logs, and is designed to scale with your growing business.

Data visualization using built-in Amazon OpenSearch Service support

Simplify data visualization using built-in Amazon OpenSearch Service support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana.

AWS Solutions Implementation overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.

Click to enlarge

Centralized Logging architecture

The Centralized Logging solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

Log ingestion: Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. CloudWatch Logs subscription filters can be configured for log groups to be streamed to the Centralized Logging account. An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
Log indexing: A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon OpenSearch Service document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
Visualization: Amazon OpenSearch Service and Kibana provide data visualization and exploration support. An Amazon OpenSearch Service domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon OpenSearch Service cluster and Kibana dashboard.