Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination. It is most often used as a data pipeline for Elasticsearch, an open-source analytics and search engine. Because of its tight integration with Elasticsearch, powerful log processing capabilities, and over 200 pre-built open-source plugins that can help you easily index your data, Logstash is a popular choice for loading data into Elasticsearch.
Easily load unstructured data
Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.
Logstash offers pre-built filters, so you can readily transform common data types, index them in Elasticsearch, and start querying without having to build custom data transformation pipelines.
Flexible plugin architecture
With over 200 plugins already available on Github, it is likely that someone has already built the plugin you need to customize your data pipeline. But if none is available that suits your requirements, you can easily create one yourself.
Getting started with Logstash on AWS
To make it easy for customers to run Elasticsearch and ingest data into it, AWS offers Amazon Elasticsearch Service, a fully managed service that delivers Elasticsearch with easy integration with Logstash. To get started, simply launch your Amazon Elasticsearch Service domain and start loading data from your Logstash server. You can try Logstash and Amazon Elasticsearch Service for free using the AWS Free Tier.
Alternative data ingestion solutions
Amazon Elasticsearch Service offers built-in integrations with Amazon Kinesis Firehose, Amazon CloudWatch Logs, and AWS IoT to help you more easily ingest data into Elasticsearch. Alternatively, you can also build your own data pipeline using open-source solutions such as Apache Kafka and Fluentd. To find out which ingestion tool is right for your use case, visit Amazon Elasticsearch Service Data Ingestion page.