What does this AWS Solution do?
A comprehensive log management and analysis strategy is mission critical, enabling organizations to understand the relationship between operational, security, and change management events and maintain a comprehensive understanding of their infrastructure. AWS customers have access to service-specific metrics and log files to gain insight into how each AWS service is operating, and many services capture additional data, such as API calls, configuration changes, and billing events. Log files from web servers, applications, and operating systems also provide valuable data, though in different formats, and in a random and distributed fashion. To effectively consolidate, manage, and analyze these different logs, many AWS customers choose to implement centralized logging solutions using either self-managed tools or AWS Partner Network (APN) offerings. These solutions provide a streamlined view of application, system, and AWS log information in the pursuit of operational excellence.
This webpage introduces an AWS solution for centralized logging and data visualization using AWS managed services. The following sections assume basic knowledge of Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), Amazon CloudWatch, Amazon Elasticsearch Service (Amazon ES), as well as a general understanding of application and system logging.
AWS Solution overview
AWS offers a centralized logging solution for collecting, analyzing, and displaying logs on AWS across multiple accounts and AWS Regions. The solution uses Amazon Elasticsearch Service (Amazon ES), a managed service that simplifies the deployment, operation, and scaling of Elasticsearch clusters in the AWS Cloud, as well as Kibana, an analytics and visualization platform that is integrated with Amazon ES. In combination with other AWS managed services, this solution offers customers a customizable, multi-account environment to begin logging and analyzing their AWS environment and applications.
The diagram below presents the centralized logging architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation templates.
Centralized Logging architecture
The primary template deploys an Amazon ES domain, which is the hardware, software, and data exposed by Amazon ES endpoints. A custom AWS Lambda function is deployed to load log data from Amazon CloudWatch to an Amazon ES domain, configured with a set of default Kibana dashboards as a starting point for data visualization.
An Amazon Cognito user pool provides Kibana dashboard user authentication. A secondary template enables customers to index logs from secondary accounts and regions on the Amazon ES domain in the primary account or region.
A demo template deploys sample logs that customers can use for testing purposes.
Centralized logging reference implementation
Access to your dashboards using Amazon Cognito
Logging capabilities beyond default AWS service logs
Data visualization using built-in Amazon ES support
Browse our portfolio of AWS-built solutions to common architectural problems.
Find AWS certified consulting and technology partners to help you get started.
Sign-up and start exploring our services.