What does this AWS Solutions Implementation do?

The Centralized Logging solution enables organizations to collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. AWS services generate log data, such as audit logs for access, configuration changes, and billing events. In addition to AWS log data; web servers, applications, and operating systems all generate log files in various formats. Consolidating, managing, and analyzing these different log files is challenging to customers.

This solution contains a suite of infrastructure services that deploy a centralized logging solution. You can collect Amazon CloudWatch Logs from multiple accounts and AWS Regions. This solution uses Amazon Elasticsearch Service (Amazon ES) and Kibana, an analytics and visualization platform that is integrated with Amazon ES, that results in a unified view of all the log events. In combination with other AWS managed services, this solution provides you with a turnkey environment to begin logging and analyzing your AWS environment and applications.

AWS Solutions Implementation overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.

Centralized Logging | Architecture Diagram
 Click to enlarge

Centralized Logging architecture

The Centralized Logging solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

  • Log ingestion: Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. CloudWatch Logs subscription filters can be configured for log groups to be streamed to the Centralized Logging account. An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
  • Log indexing: A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon Elasticsearch Service (Amazon ES) domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon ES document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
  • Visualization: Amazon ES and Kibana provide data visualization and exploration support. An Amazon ES domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon ES cluster and Kibana dashboard.

Centralized Logging

Version 4.0.0
Last updated: 12/2020
Author: AWS

Estimated deployment time: 30 min

Use the button below to subscribe to solution updates.

Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.  

Did this Solutions Implementation help you?
Provide feedback 

Features

Centralized logging reference implementation

Deploy the Centralized Logging solution using AWS CloudFormation. The CloudFormation template automatically launches and configures the components necessary to upload log files from multiple accounts and AWS Regions to Amazon ES for analysis and visualization in a customizable, user-friendly dashboard.

Access to your dashboards using Amazon Cognito

Control access to your dashboards using Amazon Cognito to simplify authentication to Amazon ES.

Logging capabilities beyond default AWS service logs

Extend your logging capabilities beyond default AWS service logs. This flexible solution includes examples for capturing host-level log files and VPC flow logs, and is designed to scale with your growing business.

Data visualization using built-in Amazon ES support

Simplify data visualization using built-in Amazon ES support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana.
Solving with Solutions: Centralized Logging
Build icon
Deploy a Solution yourself

Browse our library of AWS Solutions Implementations to get answers to common architectural problems.

Learn more 
Find an APN partner
Find an APN Partner

Find AWS certified consulting and technology partners to help you get started.

Learn more 
Explore icon
Explore Solutions Consulting Offers

Browse our portfolio of Consulting Offers to get AWS-vetted help with solution deployment.

Learn more