Build a centralized log analytics platform with Amazon OpenSearch Service on AWS in 20 minutes
Overview
Centralized Logging with OpenSearch helps organizations collect, ingest, and visualize log data from various sources using Amazon OpenSearch Service. This solution provides a web-based console, which you can use to create log ingestion pipelines with a few clicks. Log ingestion pipelines include log collection agent deployment, log enrichment without writing codes, buffer layer creation, and OpenSearch index configuration. After logs are stored in OpenSearch Service, the solution automatically generates ready-to-use dashboards for analyzing AWS service logs and application logs in different formats (for example, Nginx, JSON, and Spring Boot). In combination with other AWS services, this solution provides you with a turnkey environment to begin logging and monitoring your AWS applications.
What's new | November 2023
- Added Light Engine to provide an Amazon Athena-based serverless and cost-effective log analytics engine
- Added Amazon OpenSearch Ingestion to provide more log processing capabilities
To find out about other new features, refer to the Revisions page.
Benefits
Use a web console from your AWS account to ingest both application and AWS service logs, then analyze the logs with visualization dashboards.
Serverless technologies with built-in high availability and a pay-for-use billing model reduces the need for infrastructure management, allowing you to focus more on building log analytics for your business.
The solution is open sourced and free for commercial use. You pay only for the AWS usage. You can take the source code as a reference to make your own implementation that fits your needs.
Technical details
The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying Amazon CloudFormation template.
Step 1
Amazon CloudFront distributes the frontend web UI assets hosted in an Amazon Simple Storage Service (Amazon S3) bucket.
Step 2
Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.
Step 3
AWS AppSync provides the backend GraphQL APIs.
Step 4
Amazon DynamoDB stores the solution-related information as a backend database.
Step 5
AWS Lambda interacts with other AWS services to process core logic of managing log pipelines or log agents and obtains information updated in DynamoDB tables.
Step 6
AWS Step Functions orchestrates on-demand AWS CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate AWS resources and are used to collect and process logs and ingest them into Amazon OpenSearch Service for further analysis and visualization.
Step 7
Service Log Pipeline or Application Log Pipeline are provisioned on demand via the Centralized Logging with OpenSearch console.
Step 8
AWS Systems Manager and Amazon EventBridge manage log agents for collecting logs from application servers, such as installing log agents (Fluent Bit) for application servers and monitoring the health status of the agents.
Step 9
Amazon EC2 or Amazon EKS installs Fluent Bit agents and uploads log data to the application log pipeline.
Step 10
Application log pipelines read, parse, and process application logs and ingest them into Amazon OpenSearch Service domains or Light Engine.
Step 11
Service log pipelines read, parse, and process AWS service logs and ingest them into Amazon OpenSearch Service domains or Light Engine.
Step 1
Amazon CloudFront distributes the frontend web UI assets hosted in an Amazon Simple Storage Service (Amazon S3) bucket.
Step 2
Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.
Step 3
AWS AppSync provides the backend GraphQL APIs.
Step 4
Amazon DynamoDB stores the solution-related information as a backend database.
Step 5
AWS Lambda interacts with other AWS services to process core logic of managing log pipelines or log agents and obtains information updated in DynamoDB tables.
Step 6
AWS Step Functions orchestrates on-demand AWS CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate AWS resources and are used to collect and process logs and ingest them into Amazon OpenSearch Service for further analysis and visualization.
Step 7
Service Log Pipeline or Application Log Pipeline are provisioned on demand via the Centralized Logging with OpenSearch console.
Step 8
AWS Systems Manager and Amazon EventBridge manage log agents for collecting logs from application servers, such as installing log agents (Fluent Bit) for application servers and monitoring the health status of the agents.
Step 9
Amazon EC2 or Amazon EKS installs Fluent Bit agents and uploads log data to the application log pipeline.
Step 10
Application log pipelines read, parse, and process application logs and ingest them into Amazon OpenSearch Service domains or Light Engine.
Step 11
Service log pipelines read, parse, and process AWS service logs and ingest them into Amazon OpenSearch Service domains or Light Engine.
Related content
A new capability to search, analyze, and correlate cross-account telemetry data stored in CloudWatch such as metrics, logs, and traces.
Launch with an existing VPC in AWS Regions
Launch with a new VPC in AWS China Regions
Launch with an existing VPC in China Regions