AWS Big Data Blog

Category: Amazon OpenSearch Service

Introducing Cold Storage for Amazon OpenSearch Service

Log analytics is the most popular use case for Amazon OpenSearch Service, and with the modern-day advent of the architectural tenet to log everything all the time, it can be a challenge to store and analyze this exponential data growth effectively with a minimal price-to-performance tag. With a proliferating number of applications, log volume to […]

Use Grok patterns in AWS Glue to process streaming data into Amazon Elasticsearch Service

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. Recently, we launched AWS Glue custom connectors for Amazon OpenSearch Service, which provides the capability to ingest data into Amazon OpenSearch Service with just a few clicks. You can now use Amazon OpenSearch Service as a data store for your […]

Migrate data into Amazon ES using remote reindex

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. Amazon OpenSearch Service recently launched support for remote reindexing. This feature adds the ability to copy data to an Amazon OpenSearch Service domain from self-managed Elasticsearch running on-premises, self-managed on Amazon Elastic Compute Cloud (Amazon EC2) on AWS, or another […]

Introducing Auto-Tune in Amazon ES

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. Today we announced Auto-Tune in Amazon OpenSearch Service, an innovation undertaken to automatically optimize resources in Elasticsearch clusters to improve its performance and availability. Auto-Tune gives us a unique opportunity of applying our learnings from operating clusters at cloud scale […]

Export and import Kibana dashboards with Amazon ES

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. Kibana is a popular open-source visualization tool designed to work with Elasticsearch. Amazon OpenSearch Service provides an installation of Kibana with every Amazon OpenSearch Service domain. Users of Kibana can create visualizations and add them into a dashboard. As organizations […]

Build a data lake using Amazon Kinesis Data Streams for Amazon DynamoDB and Apache Hudi

August 30, 2023: Amazon Kinesis Data Analytics has been renamed to Amazon Managed Service for Apache Flink. Read the announcement in the AWS News Blog and learn more. July 2023: This post was reviewed for accuracy. Amazon DynamoDB helps you capture high-velocity data such as clickstream data to form customized user profiles and online order […]

Here is a component overview:

Getting started with Trace Analytics in Amazon OpenSearch Service

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.  Updated May 11, 2021. See the release notes below for more details. Trace Analytics is now available for Amazon OpenSearch Service domains running versions 7.9 or later. Developers and IT Ops teams can use this feature to troubleshoot performance and […]

Using pipes to explore, discover and find data in Amazon OpenSearch Service with Piped Processing Language

System developers, DevOps engineers, support engineers, site reliability engineers (SREs), and IT managers make sure that the underlying infrastructure powering the applications and systems within an organization is available, reliable, secure, and scalable. To achieve these goals, you need to perform a fast and deep analysis on the underlying logs, monitoring, and observability data. Amazon […]

A deep dive into high-cardinality anomaly detection in Elasticsearch

In May 2020, we announced the general availability of real-time anomaly detection for Elasticsearch. With that release we leveraged the Random Cut Forest (RCF) algorithm to identify anomalous behaviors in the multi-dimensional data streams generated by Elasticsearch queries. We focused on aggregation first, to enable our users to quickly and accurately detect anomalies in their […]

Automating Index State Management for Amazon ES

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. When it comes to time-series data, it’s more common to access new data over existing data, such as the last 4 hours or 1 day. Often, application teams are tasked with maintaining multiple indexes for diverse data workloads, which brings […]