With Amazon OpenSearch Service, choose from a selection of open-source engine options. You can deploy and run the latest versions of OpenSearch, as well as 19 versions of ALv2 Elasticsearch (7.10 and earlier). The service also includes visualization capabilities with OpenSearch Dashboards and Kibana (7.10 and earlier).
Deployment and management
Setup and configuration: Getting started with Amazon OpenSearch Service is easy. You can set up and configure your Amazon OpenSearch Service cluster using the AWS Management Console or a single API call through the AWS Command Line Interface (CLI). You can specify the number of instances, instance types, storage options, and modify or delete existing clusters at any time.
In-place upgrades: Amazon OpenSearch Service makes it easy to upgrade your OpenSearch and Elasticsearch clusters (up to version 7.10) to newer versions without any downtime, using in-place version upgrades. In-place upgrades eliminates the hassle of taking a manual snapshot, restoring it to a cluster running the newer version, and updating all your endpoint references.
Event monitoring and alerting: Amazon OpenSearch Service provides built-in event monitoring and alerting, enabling you to monitor the data stored in your cluster and automatically send notifications based on pre-configured thresholds. Built using the OpenSearch alerting plugin, this feature lets you configure and manage alerts using your Kibana or OpenSearch Dashboards interface and the REST API. You can receive notifications via custom webhooks, Slack, Amazon Simple Notification Service (SNS), and Amazon Chime. You can also view cluster health metrics including number of instances, cluster health, searchable documents, CPU, and memory, as well as disk utilization for data and master nodes through Amazon CloudWatch, at no additional charge.
Support for multiple query languages: With Amazon OpenSearch Service, there’s no need for OpenSearch query domain-specific language (DSL) proficiency. Write SQL queries with OpenSearch SQL or use the OpenSearch Piped Processing Language (PPL), a query language that lets you use pipe (|) syntax, to explore, discover, and query your data. OpenSearch Dashboards also includes a SQL and PPL workbench.
Integration with open source tools: Amazon OpenSearch Service offers built-in OpenSearch Dashboards and Kibana (Elasticsearch version 7.10 and previous) and integrates with Logstash, so you can ingest and visualize your data using the open source tools you prefer. Perform trace analytics with Amazon OpenSearch Service’s support for the open source OpenTelemetry standard and continue to use your existing code with direct access to Elasticsearch APIs and plugins such as Kuromoji, Phonetic Analysis, Ingest Processor Attachment, Ingest User Agent Processor, and Mapper Murmur3.
Security: With Amazon OpenSearch Service, you can securely connect your applications to your managed Elasticsearch (version 7.10 and previous) or OpenSearch environment from your Amazon Virtual Private Cloud (VPC) or via the public Internet, configuring network access using VPC security groups or IP-based access policies. You can also securely authenticate users and control access using Amazon Cognito, AWS Identity and Access Management (IAM), or basic authentication with a username and password. Amazon OpenSearch Service leverages the OpenSearch security plugin, enabling you to define granular permissions for indices, documents, or fields. You can also extend Kibana with read-only views and secure multi-tenant support. Amazon OpenSearch Service also supports built-in encryption for data at-rest and in-transit, so you can protect your data when it is stored in your domain or in automated snapshots and transferring between nodes in your domain. Amazon OpenSearch Service is HIPAA-eligible and compliant with PCI DSS, SOC, ISO, and FedRAMP standards, making it easy for you to build applications that meet compliance requirements.
Serverless: Automatically provision and continually adjust to get fast data ingestion rates and millisecond response times during changing usage patterns and demand with Amazon OpenSearch Serverless.
Hot storage allows for fast retrieval of frequently accessed data. UltraWarm is a warm storage tier that complements Amazon OpenSearch Service’s hot storage tier by providing less expensive storage for older and less-frequently accessed data while still providing an interactive querying experience. UltraWarm stores data in Amazon S3 and uses custom, highly-optimized nodes, purpose-built on the AWS Nitro System, to cache, pre-fetch, and query that data quickly.
With UltraWarm, you can retain up to 3 PB of data in a single Amazon OpenSearch Service cluster while reducing cost per GB by nearly 90% compared to the hot storage tier. You can also easily query and visualize the data in your Kibana (version 7.10 and previous) or OpenSearch Dashboards interface. Analyze both your recent (weeks) and historical (months or years) log data without spending hours or days restoring archived logs.
Q. What is UltraWarm?
UltraWarm is a fully-managed, low-cost, warm storage tier for Amazon OpenSearch Service. It is compatible with OpenSearch, Elasticsearch (until version 7.10), OpenSearch Dashboards, and Kibana (until version 7.10), enabling you to analyze data using the same tools that Amazon OpenSearch Service provides today. UltraWarm seamlessly integrates with Amazon OpenSearch Service’s existing features such as integrated alerting, SQL querying, and more.
Q. Why should I use UltraWarm?
UltraWarm enables you to cost effectively expand the data you want to analyze on Amazon OpenSearch Service gaining valuable insights on data that previously may have been deleted or archived. With UltraWarm, you can now economically retain more of your data to interactively analyze it whenever you want.
Q. How does UltraWarm relate to/work with Amazon OpenSearch Service?
Amazon OpenSearch Service supports two integrated storage tiers, hot and UltraWarm. The hot tier is powered by data nodes which are used for indexing, updating, and providing the fastest access to data. UltraWarm nodes complement the hot tier by providing low cost, read-only tier for older and less-frequently accessed data.
Q. Why does UltraWarm only need primary data for durability?
UltraWarm uses Amazon Simple Storage Service (Amazon S3) for storage, which is designed for 99.999999999 percent durability, and removes the need to configure an Elasticsearch replica for your warm data. Additionally, if you have more than one UltraWarm node, in the event of a node failure, the other UltraWarm nodes will automatically access the data as needed.
Q. How much data can I store in UltraWarm?
UltraWarm supports up to 3 PB of primary data. UltraWarm is designed to allow you to fully utilize 100% of this storage and because UltraWarm stores data on S3 for durability, you do not need to use additional storage for Elasticsearch replicas.
Q. What are the performance characteristics of UltraWarm?
UltraWarm delivers an interactive experience in OpenSearch Dashboards and Kibana by implementing granular I/O caching, prefetching, and query engine optimizations to provide similar performance to high-density instances using local storage.
Q. How can I start using UltraWarm?
To get started with UltraWarm, create a new Amazon OpenSearch Service domain with UltraWarm enabled via the console, CLI, or APIs. Once your domain is created you can move data from hot to UltraWarm using the OpenSearch/Elasticsearch APIs. Learn more.
Cold storage is the lowest-cost storage option for Amazon OpenSearch Service, which allows you to retain infrequently accessed data in Amazon S3 and only pay for compute when you need it. Cold storage builds on UltraWarm, which provides specialized nodes that store data in Amazon S3 and uses a sophisticated caching solution to provide an interactive experience. By decoupling compute resources from storage, cold storage lets you retain any amount of data in your Amazon OpenSearch Service domain while reducing cost per GB to near Amazon S3 storage prices. Detach historical or infrequently accessed warm data while not in use and free up compute to help lower costs. Discover and selectively attach your cold data to your domain’s UltraWarm nodes in seconds with your choice of a Kibana (version 7.10 and previous) or OpenSearch Dashboards interface and easy-to-use APIs. With cold storage, you can query the attached cold data with a similar interactive experience and performance as your warm data.
OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.
Cold storage FAQs
Q. What is cold storage?
Cold storage is a fully-managed lowest cost storage tier for Amazon OpenSearch Service that makes it easy for you to securely store and analyze your historical logs on-demand. Cold storage enables you to fully detach storage from compute when they are not actively performing analysis of their data and allows you to keep your data readily available at low cost. Cold storage data is available within the Amazon OpenSearch Service domain via your UltraWarm nodes. Cold storage seamlessly integrates with OpenSearch and OpenSearch Dashboards, as well as Elasticsearch (version 7.9, 7.10) and Kibana (version 7.9, 7.10). It enables you to analyze data using the same tools that Amazon OpenSearch Service provides today.
Q. Why should I use cold storage?
Cold storage enables you to cost effectively expand the data you want to analyze on Amazon OpenSearch Service and gain valuable insights on data that previously may have been deleted or archived. Cold storage is a great fit if you have the need to do research or forensic analysis on your older data and you want to use all of the capabilities of Amazon OpenSearch Service to do so, at an affordable price. Cold storage is built for scale and is backed by Amazon S3. Find and discover the data you need, attach it to the UltraWarm nodes in your cluster, and make it available for analysis in seconds. Attached cold data is subject to the existing fine-grained access control policies that limit access at the index, document, and field level.
Q. How does cold storage relate to/work with Amazon OpenSearch Service?
With cold storage, Amazon OpenSearch Service supports three integrated storage tiers: hot, UltraWarm, and cold. The hot tier is used for indexing, updating, and providing the fastest access to data. UltraWarm provides a seamless extension of the hot tier by providing compute nodes that provide a highly performant interactive experience for data that is durably stored in Amazon S3 and needs to be persistently available, currently supporting up to 3PB of data in a single domain. With cold storage, you can now detach indices from UltraWarm while not in use and free up compute to help lower costs. With the new cold storage APIs and OpenSearch Dashboards and Kibana interface, you can discover indices based on index patterns and data timestamps to easily find what you need for analysis. That data can then be attached to the domain and ready for analysis in seconds. When you are done with analysis, simply detaching the data then frees up your compute again.
Q. How much data can I store in cold storage?
Cold storage is built for scale. While the storage limits for hot and warm data remain at 3PB, you can store any amount of data in cold storage.
Q. What are the performance characteristics of cold storage?
Cold storage builds on UltraWarm, which provides specialized nodes that store data in Amazon S3 and uses a sophisticated caching solution to provide an interactive experience. Cold data must first be attached to the UltraWarm nodes of your Amazon OpenSearch Service domain. Once attached, queries on this data are powered by existing UltraWarm nodes offering the same performance as your warm data. Attaching cold indices to your domain takes seconds if there is sufficient UltraWarm capacity available for the requested data. If you need additional capacity, UltraWarm data nodes must be added, which can take up to a few minutes.
OpenSearch Service provides real-time document search capabilities that go beyond database search. This fully managed service uses the OpenSearch engine for search. OpenSearch is a full-featured, Lucene-based, portable, platform-agnostic open-source search engine supporting keyword search, natural language search, synonyms, multiple languages, and more. Core search capabilities:
- Acquires data from a database or content management system, a web or intranet crawler, or a streaming service
- Provides search APIs to build a frontend on top of the search services
- Powers searches across many attributes
- Finds new documents that match a set of saved queries with prospective search (percolation)
- Assesses usage patterns and performs capacity planning and cost prediction with OpenSearch Service monitoring capabilities
- Uses built-in machine learning (ML) algorithms for k-nearest neighbors (k-NN) search to accomplish vector search, similarity search, semantic search, and more
- Uses built-in ML algorithm for Learning to Rank to calculate relevance scores.
- Uses multiple query languages, including SQL
Enable your Security Operations (SecOps) teams to detect potential threats quickly while having the tools to help with security investigations, all with low data retention costs. Secure your business data and rapidly detect potential security threats. OpenSearch Service provides out-of-the-box support for over 2200 open source Sigma security rules to detect potential security threats by filtering through the security findings. You can even customize or use default Sigma rules to rapidly detect potential security threats and send alerts to a pre-selected destination. Use out-of-the-box support for multiple log sources including Windows, Netflow, AWS CloudTrail, DNS, and more.
Q: What is security analytics?
OpenSearch security analytics is designed to help investigate, detect, analyze, and respond to security threats that could jeopardize the operations of business critical functions. These threats include the potential exposure of confidential data, cyber attacks, and other adverse security events. It includes the tools and features necessary for defining detection parameters, generating alerts, and responding effectively to potential threats.
Q: What type of security logs does security analytics support?
We currently support 8 log types including Netflow, DNS logs, Apache access logs, Windows logs, AD/LDAP logs, Linux system logs, AWS CloudTrail logs and Amazon S3 access logs
Q: How do I send these security logs to OpenSearch?
You can use your existing ingestion pipelines that send JSON formatted data to OpenSearch.
Q: Does security analytics provide security rules out of the box?
Yes, OpenSearch security analytics packages over 2200 Sigma security rules for out of the box use with different types of security detectors. These rules are preselected once you provide minimal configuration about the log source.
Q: Can I create my own custom rules?
Yes, custom rules can be added for the supported log types above. These rules need to be in a Sigma rule format and can be imported into OpenSearch before using with a security detector.
Q: Do I need to convert the logs to a specific format or schema?
Yes, the logs must be in JSON format. We recommend to send them in ECS (Elastic Common Schema) format
Q: Do I need to pay additional licensing fees to use security analytics?
OpenSearch security analytics is available to you for no additional cost or licensing fees. You pay the same cost as you would to ingest other data into OpenSearch Service.
Q: What version of OpenSearch Service supports security analytics?
Security Analytics comes preinstalled with OpenSearch Service running OpenSearch version 2.5 or higher.
Q: Are there any differences between OpenSearch security analytics and Amazon Security Lake?
Amazon Security Lake automatically centralizes security data from cloud, on-premises, and custom sources into a purpose-built data lake stored in your account. This aggregated data is normalized into a common format, stored in S3 buckets. This data can be ingested into OpenSearch Service, that allows you to visualize, query, create reports on the same. Security analytics provides a security rules engine that can help you to detect and alert on potential security events, as well as help you to correlate them to help with your investigation.
Q: Can I use OpenSearch security analytics with Amazon Security Lake?
Yes, you can bring additional logs from Security Lake into OpenSearch and create a detector to run relevant rules on the ingested logs.
OpenSearch Service feature details:
Learn more about Amazon OpenSearch Service pricing