What does this AWS Solution do?

A comprehensive log management and analysis strategy is mission critical, enabling organizations to understand the relationship between operational, security, and change management events and maintain a comprehensive understanding of their infrastructure. AWS customers have access to service-specific metrics and log files to gain insight into how each AWS service is operating, and many services capture additional data, such as API calls, configuration changes, and billing events. Log files from web servers, applications, and operating systems also provide valuable data, though in different formats, and in a random and distributed fashion. To effectively consolidate, manage, and analyze these different logs, many AWS customers choose to implement centralized logging solutions using either self-managed tools or AWS Partner Network (APN) offerings. These solutions provide a streamlined view of application, system, and AWS log information in the pursuit of operational excellence.

This webpage introduces an AWS solution for centralized logging and data visualization using AWS managed services. The following sections assume basic knowledge of Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), Amazon CloudWatch, Amazon Elasticsearch Service (Amazon ES), as well as a general understanding of application and system logging.

AWS Solution overview

AWS offers a centralized logging solution for collecting, analyzing, and displaying logs on AWS across multiple accounts and AWS Regions. The solution uses Amazon Elasticsearch Service (Amazon ES), a managed service that simplifies the deployment, operation, and scaling of Elasticsearch clusters in the AWS Cloud, as well as Kibana, an analytics and visualization platform that is integrated with Amazon ES. In combination with other AWS managed services, this solution offers customers a customizable, multi-account environment to begin logging and analyzing their AWS environment and applications.

The diagram below presents the centralized logging architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.

centralized-logging-architecture
 Click to enlarge

Centralized Logging architecture

The primary template deploys an Amazon ES domain, which is the hardware, software, and data exposed by Amazon ES endpoints. A custom AWS Lambda function is deployed to load log data from Amazon CloudWatch to an Amazon ES domain, configured with a set of default Kibana dashboards as a starting point for data visualization.

An Amazon Cognito user pool provides Kibana dashboard user authentication. A secondary template enables customers to index logs from secondary accounts and regions on the Amazon ES domain in the primary account or region. 

A demo template deploys sample logs that customers can use for testing purposes.

Centralized Logging

Version 2.2.0
Last updated: 11/2018
Author: AWS

Estimated deployment time: 30 min

Features

Centralized logging reference implementation

Deploy a centralized logging solution using AWS CloudFormation. The CloudFormation template will automatically launch and configure the components necessary to upload log files from multiple accounts and AWS Regions to Amazon ES for analysis and visualization in a customizable, user-friendly dashboard.

Access to your dashboards using Amazon Cognito

Control access to your dashboards using Amazon Cognito to simplify authentication to Amazon ES.

Logging capabilities beyond default AWS service logs

Extend your logging capabilities beyond default AWS service logs. This flexible solution includes examples for capturing host-level log files and VPC flow logs, and is designed to scale with your growing business.

Data visualization using built-in Amazon ES support

Simplify data visualization using built-in Amazon ES support for Kibana, including a default set of preconfigured dashboards that give you a first glimpse into the customization capabilities of Kibana.
Product-Page_Standard-Icons_01_Product-Features_SqInk
Explore all AWS Solutions

Browse our portfolio of AWS-built solutions to common architectural problems.

Learn more 
Next-Steps-Icon_Find-a-Partner-B
Find a Partner

Find AWS certified consulting and technology partners to help you get started.

Learn more 
Product-Page_Standard-Icons_03_Start-Building_SqInk
Start building in the console

Sign-up and start exploring our services.

Get started