AWS Solutions Library

Centralized Logging on AWS

What does this AWS Solution do?

The Centralized Logging on AWS solution helps organizations collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. This solution consolidates, manages, and analyzes log files from various sources. You can collect Amazon CloudWatch Logs from multiple accounts and AWS Regions.

This solution uses Amazon OpenSearch Service and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, that results in a unified view of all the log events. In combination with other AWS managed services, this solution provides you with a turnkey environment to begin logging and analyzing your AWS environment and applications.

Benefits

Use this solution out-of-the box, or build and customize

blank

The AWS CloudFormation template automatically launches and configures the components necessary to upload log files from multiple accounts and AWS Regions to Amazon OpenSearch Service for analysis and visualization in a customizable, user-friendly dashboard.

Access to your dashboards using Amazon Cognito

blank

Control access to your dashboards using Amazon Cognito to simplify authentication to Amazon OpenSearch Service.

Logging capabilities beyond default AWS service logs

blank

Extend your logging capabilities beyond default AWS service logs. This flexible solution includes examples for capturing host-level log files and VPC flow logs, and is designed to scale with your growing business.

Data visualization using built-in Amazon OpenSearch Service support

Simplify data visualization using built-in Amazon OpenSearch Service support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana.

AWS Solution overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.

Centralized Logging | Architecture Diagram

Centralized Logging on AWS architecture

The Centralized Logging on AWS solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

Log ingestion: Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. CloudWatch Logs subscription filters can be configured for log groups to be streamed to the Centralized Logging on AWS account. An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
Log indexing: A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon OpenSearch Service domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon OpenSearch Service document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
Visualization: Amazon OpenSearch Service and Kibana provide data visualization and exploration support. An Amazon OpenSearch Service domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon OpenSearch Service cluster and Kibana dashboard.

Show less