AWS Cloud
Get started with Log Analytics

Servers, applications, websites, and connected devices generate discrete, time-stamped records of events called logs. Processing and analyzing these logs to gain actionable insights is called log analytics. Early log analytics solutions were designed for IT operational intelligence use cases, such as root cause analysis and infrastructure monitoring. Over time, log analytics solutions have incorporated additional data sources, machine learning, and other analytical techniques to enable additional use cases in application performance management (APM), security intelligence and event management (SIEM), and business analytics.

The volume and importance of log data is growing rapidly. In the past, IT professionals monitored applications by installing APM agents to collect metrics. But, now as companies migrate many of their applications from on-premises to the cloud, they increasingly rely on system logs for monitoring and root cause analysis. In addition to the cloud migration, businesses are also moving their architectures to micro-services to enable faster deployment and scaling. Micro-service architectures break down applications into several components, each of which generates its own log and metrics data, thus increasing the number of entities producing system log data. Companies want to capture and centralize all this data so they can understand the relationship between operational, security, and change management events and maintain a comprehensive view of their infrastructure.

AWS customers have access to service-specific metrics and log files to gain insights into how each AWS service is operating. Many services capture additional data, such as API calls, configuration changes, and billing events. In addition, log files from web servers, applications, and operating systems provide valuable data, though in different formats, and in a random and distributed fashion. To effectively consolidate, manage, and analyze these different logs, many AWS customers choose to implement centralized logging solutions using either self-managed tools or AWS Partner Network (APN) offerings. These solutions provide a streamlined view of application, system, and AWS log information in the pursuit of operational excellence.

There are a few different ways in which you can build log analytics solutions on AWS. We will highlight two popular approaches in the following. 


You can use Amazon Elasticsearch Service (Amazon ES), a fully managed service that makes it easy to deploy, scale, and use Elasticsearch on AWS, along with Kibana, an analytics and visualization platform that is integrated with Amazon ES. In combination with other AWS services, this solution offers customers a highly available, turnkey environment to begin logging and analyzing their AWS environment and applications.

The diagram below presents the centralized logging architecture. To learn more, see the Centralized Logging Solution Brief.


You can use Amazon Kinesis, which makes it easy to stream, process, and analyze real-time data, along with AWS CloudTrail, which is integrated with Kinesis through Amazon CloudWatch events triggers. This solution enables you build a serverless solution to monitor your applications in real time.

The diagram below presents the real-time application monitoring architecture. To learn more, check out this hands-on tutorial.

IT Operational Intelligence

You can gain visibility into distributed architectures for three main use cases: log forensics (also known as root cause analysis), infrastructure monitoring, and application monitoring. Log forensics is a frequent use case for Amazon Elasticsearch Service. Customers, such as Ancestry, use Amazon Elasticsearch Service to provide logging-as-a-service for individual teams to debug issues in their applications. Users perform full-text search, drill downs, aggregations, and visualizations to find the source of IT issues. This process can be enhanced with correlation technology that brings relevant information forward to reduce the mean time to repair (MTTR), a key metric for IT Ops teams.

Application Performance Monitoring

Growing datasets, increasing complexity, and the need to provide an always-on experience for end users are pushing customers to find ways to identify issues as they occur using anomaly detection, real-time dashboards, and automated alerts. For example, log analytics application can monitor a data stream and alert when it has deviated from normal. Netflix uses Amazon Kinesis to monitor the communications between all of its applications so it can detect and fix issues quickly, ensuring high service uptime and availability to its customers. Read the case study »

Security Intelligence and Event Management

SIEM use cases perform real-time intrusion detection, user behavior analytics, and fraud detection. The challenge with security is having the background information – user profiles and threat intelligence – to identify security events. But advances in machine learning are lowering this barrier. These solutions pre-process data to identify sequences of events and apply machine learning algorithms to identify anomalous trends and predict a potential security problem.

Business Analytics

Business analytics can be divided into batch and real-time analytics. Batch analytics is the more classical definition of business intelligence, such as finance or accounting jobs that are well suited for a relational database. Log analytics platforms can perform simple business intelligence queries. But the value for log analytics systems lies in real-time business analytics, due to the streaming nature of log and machine data. Customers use Amazon Kinesis to stream, process, and load data in real-time to Amazon Elasticsearch Service.

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, data lakes and data warehouses, or build your own real-time applications using this data. Amazon Kinesis enables you to process and analyze data as it arrives and respond in real-time instead of having to wait until all your data is collected before the processing can begin.

Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch for log analytics, full text search, application monitoring, and more. Amazon Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. The service offers built-in integrations with Kibana, Logstash, and AWS services including Amazon Kinesis Firehose, AWS Lambda, and Amazon CloudWatch so that you can go from raw data to actionable insights quickly.

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources.


It's easy to get started with Amazon Kinesis. Just sign up for an AWS account, or if you already have one, sign in to the AWS Management Console, and launch Amazon Kinesis.

 

Get Started with Amazon Kinesis