What is the ELK stack?
The ELK stack is an acronym used to describe a stack that comprises of three popular projects: Elasticsearch, Logstash, and Kibana. Often referred to as Elasticsearch, the ELK stack gives you the ability to aggregate logs from all your systems and applications, analyze these logs, and create visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics, and more.
E = Elasticsearch
Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Support for various languages, high performance, and schema-free JSON documents makes Elasticsearch an ideal choice for various log analytics and search use cases. Learn more »
On January 21, 2021, Elastic NV announced that they would change their software licensing strategy and not release new versions of Elasticsearch and Kibana under the permissive Apache License, Version 2.0 (ALv2) license. Instead, new versions of the software will be offered under the Elastic license, with source code available under the Elastic License or SSPL. These licenses are not open source and do not offer users the same freedoms. To ensure that the open source community and our customers continue to have a secure, high-quality, fully open source search and analytics suite, we introduced the OpenSearch project, a community-driven, ALv2 licensed fork of open source Elasticsearch and Kibana. The OpenSearch suite consists of a search engine, OpenSearch, and a visualization and user interface, OpenSearch Dashboards.
L = Logstash
Logstash is an open-source data ingestion tool that allows you to collect data from a variety of sources, transform it, and send it to your desired destination. With pre-built filters and support for over 200 plugins, Logstash allows users to easily ingest data regardless of the data source or type. Learn more »
K = Kibana
Kibana is a data visualization and exploration tool for reviewing logs and events. Kibana offers easy-to-use, interactive charts, pre-built aggregations and filters, and geospatial support and making it the preferred choice for visualizing data stored in Elasticsearch. Learn more »
The ELK Stack fulfills a need in the log analytics space. As more and more of your IT infrastructure move to public clouds, you need a log management and analytics solution to monitor this infrastructure as well as process any server logs, application logs, and clickstreams. The ELK stack provides a simple yet robust log analysis solution for your developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring – at a fraction of the price.
Why is the ELK stack so popular?
The ELK Stack is popular because it fulfills a need in the log analytics space. As more and more of your IT infrastructure move to public clouds, you need a log management and analytics solution to monitor this infrastructure as well as process any server logs, application logs, and clickstreams. The ELK stack provides a simple yet robust log analysis solution for your developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring – at a fraction of the price.
The ELK stack – Choosing the right option
You can choose to deploy and manage the ELK stack yourself with Apache 2.0 licensed versions of Elasticsearch and Kibana (up until version 7.10.2) or self-manage an open source alternative to the ELK stack with OpenSearch, OpenSearch Dashboards, and Logstash. But, would you prefer that your developers or DevOps engineers spend time on building innovative applications or on managing operational tasks such as deployment, upgrades, software installation and patching, backups, and monitoring? Also, scaling up and down to meet your business requirements or achieving security and compliance is a challenge with the self-managed option.
Or you can choose an easier, scalable, and more secure option.
Introducing Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)
Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), and visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions). The service integrates with Logstash as well as other AWS services such as Amazon Kinesis Data Firehose, Amazon CloudWatch Logs, and AWS IoT to give you the flexibility to select the data ingestion tool that meets your use case requirements.