What does this AWS Solutions Implementation do?
The Centralized Logging solution enables organizations to collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. AWS services generate log data, such as audit logs for access, configuration changes, and billing events. In addition to AWS log data; web servers, applications, and operating systems all generate log files in various formats. Consolidating, managing, and analyzing these different log files is challenging to customers.
This solution contains a suite of infrastructure services that deploy a centralized logging solution. You can collect Amazon CloudWatch Logs from multiple accounts and AWS Regions. This solution uses Amazon Elasticsearch Service (Amazon ES) and Kibana, an analytics and visualization platform that is integrated with Amazon ES, that results in a unified view of all the log events. In combination with other AWS managed services, this solution provides you with a turnkey environment to begin logging and analyzing your AWS environment and applications.
AWS Solutions Implementation overview
The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.
Centralized Logging architecture
The Centralized Logging solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.
- Log ingestion: Amazon CloudWatch Logs destinations deploy in the primary account and are created with the required permissions in each of the selected Regions. CloudWatch Logs subscription filters can be configured for log groups to be streamed to the Centralized Logging account. An optional demo AWS CloudFormation template can be deployed to generate sample CloudWatch Logs for AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, and an Amazon Elastic Compute Cloud (Amazon EC2) web server.
- Log indexing: A centralized Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are provisioned to index log events on the centralized Amazon Elasticsearch Service (Amazon ES) domain. The CloudWatch Logs destinations created to stream log events, have Kinesis Data Streams as their target. Once the log events stream to Kinesis Data Streams, the service invokes an AWS Lambda function to transform each log event to an Amazon ES document, which is then put into Kinesis Data Firehose. You can monitor Kinesis Data Firehose while it sends custom CloudWatch Logs containing detailed monitoring data for each delivery stream.
- Visualization: Amazon ES and Kibana provide data visualization and exploration support. An Amazon ES domain is created inside an Amazon VPC, preventing public access to the Kibana dashboard. Optionally, a Microsoft Windows Jumpbox Server can be launched to access the Amazon ES cluster and Kibana dashboard.
Centralized Logging
Version 4.0.0
Last updated: 12/2020
Author: AWS
Estimated deployment time: 30 min
Deployment resources
Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
Features
Centralized logging reference implementation
Access to your dashboards using Amazon Cognito
Logging capabilities beyond default AWS service logs
Data visualization using built-in Amazon ES support
Browse our library of AWS Solutions Implementations to get answers to common architectural problems.
Find AWS certified consulting and technology partners to help you get started.
Browse our portfolio of Consulting Offers to get AWS-vetted help with solution deployment.