Log analytics is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, games, and IoT. In this project, you will use Amazon Web Services to build an end-to-end log analytics solution that collects, ingests, processes, and loads both batch data and streaming data, and makes the processed data available to your users in analytics systems they are already using and in near real-time. The solution is highly reliable, cost-effective, scales automatically to varying data volumes, and requires almost no IT administration.
What you'll accomplish:
Set up a Kinesis Agent on data sources to collect data and send it continuously to Amazon Kinesis Data Firehose.
Create an end-to-end data delivery stream using Kinesis Data Firehose. The delivery stream will transmit your data from the agent to destinations including Amazon Kinesis Data Analytics, Amazon OpenSearch Service, and Amazon S3.
Process incoming log data using SQL queries in Amazon Kinesis Data Analytics.
Load processed data from Kinesis Data Analytics to Amazon OpenSearch Service to index the data.
Analyze and visualize the processed data using Kibana.
What you'll need before starting:
An AWS Account: You will need an AWS account to begin provisioning resources to host your website. Sign up for AWS.
IT Experience: You will need a basic understanding of web technologies and familiarity with SQL to successfully complete this project.
AWS Experience: No prior experience with AWS is required to successfully complete this project.
Billing Estimates:
Cost to complete project: The estimated cost to complete this project is $0.57. This cost assumes that you are within the AWS Free Tier limits, you follow the recommended configurations, and that you terminate all resources within 1 hour of completing the project. Your use case may require different configurations that can impact your bill. Use the Pricing Calculator to estimate costs tailored for your needs.
Monthly billing estimate: The total cost of building and maintaining your log analytics solution will vary depending on your usage and configuration settings. Using the default configuration recommended in this guide, it will typically cost $382.45 per month. To see a breakdown of the services used and their associated costs, see Services Used and Costs.
Learn more about streaming data solutions on Amazon Web Services (AWS).
Access developer guides and API references for more information about Amazon Kinesis.
Need more resources to get started with AWS? Visit the Getting Started Resource Center to learn more.