Applications, servers, cloud infrastructures, IoT & mobile devices, DevOps, microservices architectures – the biggest business and IT trends are helping us improve operations and customer experience like never before. But, these trends have also resulted in an explosive growth of machine-generated data, which includes logs and metrics such as user transactions, customer behavior, sensor activity, machine behavior, and security threats. This data is complex, but also the most valuable as it contains operational intelligence for IT, security, and business.
Log analytics involves searching, analyzing, and visualizing machine data generated by your IT systems and technology infrastructure to gain operational insights. Traditional data analytics tools are simply not built to handle the variety and volume of rapidly proliferating machine data. Amazon Elasticsearch Service is a fully managed service that collects, indexes, and unifies logs and metrics across your environment to give you unprecedented visibility into your applications and infrastructure. With Amazon Elasticsearch service, you get the scalability, flexibility, and security you need for the most demanding log analytics workloads.
“Ultimately, we are improving our software products and offering better service to our customers because of the real-time visibility we’re getting into log data.”
Tommy Li, Senior Software Architect, Autodesk
Why Amazon Elasticsearch Service for log analytics?
Deploy and manage with ease
Amazon Elasticsearch Service makes it simple to set up and deploy your cluster, while removing the complexity associated with management tasks, such as hardware provisioning, software installing and patching, failure recovery, backups, and monitoring, allowing you to reduce operational overhead and focus on core business requirements.
Get scale and reliability
With up to 3 PB of instance storage in a single domain, Amazon Elasticsearch Service lets you easily add or remove instances without any downtime. The service also offers built-in encryption at-rest and in-transit, user authentication, and VPC support, allowing you to keep your data secure.
Integrate easily with other AWS services
Amazon Elasticsearch Service offers built-in integrations with other AWS services such as Kinesis Data Firehose, Managed Streaming for Kafka, IoT, CloudWatch Logs, KMS, Cognito, and IAM, so you can securely ingest, analyze, and visualize data from all sources.
Lower your costs
With Amazon Elasticsearch Service, you pay only for what you use. There is no upfront fee or usage requirement. With 24x7 monitoring and AWS support, you don’t need a team of Elasticsearch experts to scale, secure, and monitor your infrastructure, resulting in lower total cost of operations.
Leading use cases
Real-time application and infrastructure monitoring
Capture and centralize all logs and metrics from your applications and IT silos to get deep visibility into your application and infrastructure stack and ensure uptime. Amazon Elasticsearch Service indexes the data, makes it available for analysis in real time, and allows you to visualize the performance metrics in real time using Kibana dashboards.
Expedia Group, one of the world’s leading travel companies, uses Amazon Elasticsearch Service for application monitoring. Amazon Elasticsearch Service enables Expedia to monitor large volumes of Docker logs cost-effectively, identify and troubleshoot issues in real-time, scale easily to accommodate additional log sources, and offload their operational overhead. Learn more »
Quickly identify the issues across your environment (servers, code) to reduce the mean-time-to-identification (MTTI) and mean-time-to-resolution (MTTR). With built-in Kibana, Amazon Elasticsearch Service lets you search through millions of events and correlate across your applications and infrastructure to quickly diagnose the root-cause of the problem, improving uptime.
Autodesk, a leading provider of 3D design and engineering software, uses AWS services including Amazon Elasticsearch Service, Amazon Kinesis Data Firehose, and Amazon Kinesis Data Analytics to build a cost-effective unified logging solution to find and fix application issues faster and improve customer experience. Learn more »
Get a real-time view of the performance of your web content and user interaction with your applications and websites including user behavior, amount of time spent, popular content, and more. Using Amazon Elasticsearch Service and Amazon Kinesis Data Firehose or Amazon Managed Streaming for Kafka, you can aggregate and analyze your clickstream logs effortlessly to gain a deeper understanding of your customers.
Hearst Corporation, a large media company, built a clickstream analytics platform using Amazon Elasticsearch Service, Amazon Kinesis Streams, and Amazon Kinesis Firehose to transmit and process 30 terabytes of data per day. With this platform, Hearst is able to make the entire data stream—from website clicks to aggregated data—available to editors in minutes.. Learn more »
Security intelligence and event management (SIEM)
Centralize and analyze events generated across your entire environment including applications, networks, and operating systems to identify any malicious or suspect activity in your network. Amazon Elasticsearch Service allows you to index the data as soon as it is ingested, allowing you to analyze data from multiple sources instantly and find and prevent threats faster.
With AWS, you can build different solutions to effectively consolidate, monitor, and analyze your log data. These solutions provide you a streamlined view of your applications, systems, and AWS log information for real-time operational intelligence.
Centralized logging using Amazon Elasticsearch Service
In combination with other AWS services, this solution powered by Amazon Elasticsearch Service provides you a highly available, turnkey environment to quickly begin logging and analyzing your AWS environment and applications. Get started with Amazon Elasticsearch Service »
The diagram below presents the centralized logging architecture. To learn more, read the centralized logging solution brief.
Real-time monitoring using Amazon Kinesis
Using Amazon Kinesis along with AWS CloudTrail and Amazon CloudWatch, this solution enables you to build a serverless solution to monitor your applications in real time. Get started with Amazon Kinesis »
The diagram below presents the real-time application monitoring architecture. To learn more, follow this hands-on tutorial.