Build a Log Analytics Solution
Collect, process, and analyze log data using Amazon Kinesis and OpenSearch Service
TUTORIAL
Overview
Log analytics is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, games, and IoT. In this project, you will use Amazon Web Services to build an end-to-end log analytics solution that collects, ingests, processes, and loads both batch data and streaming data, and makes the processed data available to your users in analytics systems they are already using and in near real-time. The solution is highly reliable, cost-effective, scales automatically to varying data volumes, and requires almost no IT administration.
What you will accomplish
- Set up a Kinesis Agent on data sources to collect data and send it continuously to Amazon Kinesis Data Firehose.
- Create an end-to-end data delivery stream using Kinesis Data Firehose. The delivery stream will transmit your data from the agent to destinations including Amazon Managed Service for Apache Flink, Amazon OpenSearch Service, and Amazon S3.
- Process incoming log data using SQL queries in Amazon Managed Service for Apache Flink.
- Load processed data from Kinesis Data Analytics to Amazon OpenSearch Service to index the data.
- Analyze and visualize the processed data using Kibana.
FAQs
- Q: How does this project architecture help me easily launch a robust log analytics solution?
With just a few clicks in the AWS Management Console, you can assemble a solution using the range of services we offer and go from raw data to real insights in minutes. Because these services are managed, you don’t have to spend time and money in planning, provisioning, and managing infrastructure. Instead, you can focus on your business problems.
- Q: Why should I run log analytics on Amazon Managed Service for Apache Flink?
Amazon Managed Service for Apache Flink is ideally suited for log analytics use cases because it makes it easy to handle unstructured data, automatically infers the structure and format of the data, and suggests a schema. With Amazon Managed Service for Apache Flink, all you need is standard SQL. You can immediately start querying the data and get real-time insights.
- Q: Besides log analytics, what other use cases can I run on Amazon Managed Service for Apache Flink?
You can use Kinesis Data Analytics in pretty much any use case where you are collecting data continuously in real-time and want to get information and insights in seconds or minutes rather than having to wait days or even weeks. In particular, Kinesis Data Analytics enables you to quickly build end-to-end stream processing applications for log analytics, clickstream analytics, Internet of Things (IoT), ad tech, games, and more. The three most common usage patterns are time-series analytics, real-time dashboards, and real-time alerts and notifications.
AWS experience
Intermediate
Time to complete
1 Hour
Cost to complete
The estimated cost to complete this project is $0.57. This cost assumes that you are within the AWS Free Tier limits, you follow the recommended configurations, and that you terminate all resources within 1 hour of completing the project. Your use case may require different configurations that can impact your bill. Use the Pricing Calculator to estimate costs tailored for your needs
Requires
Active AWS Account
Services used
Last updated
September 2021
Download the Implementation Guide
Note: Amazon Kinesis Data Analytics has been renamed to Amazon Managed Service for Apache Flink.