AWS Open Source Blog

Introducing OpenSearch

Today, we are introducing the OpenSearch project, a community-driven, open source fork of Elasticsearch and Kibana. We are making a long-term investment in OpenSearch to ensure users continue to have a secure, high-quality, fully open source search and analytics suite with a rich roadmap of new and innovative functionality. This project includes OpenSearch (derived from Elasticsearch 7.10.2) and OpenSearch Dashboards (derived from Kibana 7.10.2). Additionally, the OpenSearch project is the new home for our previous distribution of Elasticsearch (Open Distro for Elasticsearch), which includes features such as enterprise security, alerting, machine learning, SQL, index state management, and more. All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2). We invite you to check out the code for OpenSearch and OpenSearch Dashboards on GitHub, and join us and the growing community around this effort.

We welcome individuals and organizations who are users of Elasticsearch, as well as those who are building products and services based on Elasticsearch. Our goal with the OpenSearch project is to make it easy for as many people and organizations as possible to use OpenSearch in their business, their products, and their projects. Whether you are an independent developer, an enterprise IT department, a software vendor, or a managed service provider, the ALv2 license grants you well-understood usage rights for OpenSearch. You can use, modify, extend, embed, monetize, resell, and offer OpenSearch as part of your products and services. We have also published permissive usage guidelines for the OpenSearch trademark, so you can use the name to promote your offerings. Broad adoption benefits all members of the community.

We plan to rename our existing Amazon Elasticsearch Service to Amazon OpenSearch Service. Aside from the name change, customers can rest assured that we will continue to deliver the same great experience without any impact to ongoing operations, development methodology, or business use. Amazon OpenSearch Service will offer a choice of open source engines to deploy and run, including the currently available 19 versions of ALv2 Elasticsearch (7.9 and earlier, with 7.10 coming soon) as well as new versions of OpenSearch. We will continue to support and maintain the ALv2 Elasticsearch versions with security and bug fixes, and we will deliver all new features and functionality through OpenSearch and OpenSearch Dashboards. The Amazon OpenSearch Service APIs will be backward compatible with the existing service APIs to eliminate any need for customers to update their current client code or applications. Additionally, just as we did for previous versions of Elasticsearch, we will provide a seamless upgrade path from existing Elasticsearch 6.x and 7.x managed clusters to OpenSearch.

We are not alone in our commitment to OpenSearch. Organizations as diverse as Red Hat, SAP, Capital One, and Logz.io have joined us in support.

“At Red Hat, we believe in the power of open source, and that community collaboration is the best way to build software,” said Deborah Bryant, Senior Director, Open Source Program Office, Red Hat. “We appreciate Amazon’s commitment to OpenSearch being open and we are excited to see continued support for open source at Amazon.”

“SAP customers expect a unified, business-centric and open SAP Business Technology Platform,” said Jan Schaffner, SVP and Head of BTP Foundational Plane. “Our observability strategy uses Elasticsearch as a major enabler. OpenSearch provides a true open source path and community-driven approach to move this forward.”

“At Capital One, we take an open source-first approach to software development, and have seen that we’re able to innovate more quickly by leveraging the talents of developer communities worldwide,” said Nureen D’Souza, Sr. Manager for Capital One’s Open Source Program Office. “When our teams chose to use Elasticsearch, the freedoms provided by the Apache-v2.0 license was central to that choice. We’re very supportive of the OpenSearch project, as it will give us greater control and autonomy over our data platform choices while retaining the freedom afforded by an open source license.”

“At Logz.io we have a deep belief that community driven open source is an enabler for innovation and prosperity,” said Tomer Levy, co-founder and CEO of Logz.io. “We have the highest commitment to our customers and the community that relies on open source to ensure that OpenSearch is available, thriving, and has a strong path forward for the community and led by the community. We have made a commitment to work with AWS and other members of the community to innovate and enable every organization around the world to enjoy the benefits of these critical open source projects.”

Moving with the community

We are truly excited about the potential for OpenSearch to be a community endeavor, where anyone can contribute to it, influence it, and make decisions together about its future. Community development, at its best, lets people with diverse interests have a direct hand in guiding and building products they will use; this results in products that meet their needs better than anything else. It seems we aren’t alone in this interest; there’s been an outpouring of excitement from the community to drive OpenSearch, and questions about how we plan to work together.

We’ve taken a number of steps to make it easy to collaborate on OpenSearch’s development. The entire code base is under the Apache 2.0 license, and we don’t ask for a contributor license agreement (CLA). This makes it easy for anyone to contribute. We’re also keeping the code base well-structured and modular, so everyone can easily modify and extend it for their own uses.

Amazon is the primary steward and maintainer of OpenSearch today, and we have proposed guiding principles for development that make it clear that anyone can be a valued stakeholder in the project. We invite everyone to provide feedback and start contributing to OpenSearch. As we work together in the open, we expect to uncover the best ways to collaborate and empower all interested stakeholders to share in decision making. Cultivating the right governance approach for an open source project requires thoughtful deliberation with the community. We’re confident that we can find the best approach together over time.

Moving the code forward

Getting OpenSearch to this point required substantial work to remove Elastic commercial licensed features, code, and branding. The OpenSearch repos we made available today are a foundation on which everyone can build and innovate. You should consider the initial code to be at an alpha stage — it is not complete, not thoroughly tested, and not suitable for production use. We are planning to release a beta in the next few weeks, and expect it to stabilize and be ready for production by early summer (mid-2021).

The code base is ready, however, for your contributions, feedback, and participation. To get going with the repos, grab the source from GitHub and build it yourself:

Once you’ve cloned the repos, see what you can do. These repos are under active construction, so what works or doesn’t work will change from moment to moment. Some tasks you can do to help include:

Once you have OpenSearch and OpenSearch Dashboards running:

  • Test any custom plugins or code you use and report what breaks.
  • Run a sample workload and get in touch if it behaves differently from your previous setup.
  • Connect it to any external tools / libraries and find out what works as expected.
  • Put OpenSearch Dashboards in front of a typical user.

We encourage everybody to engage with the OpenSearch community. We have launched a community site at opensearch.org. Our forums are where we collaborate and make decisions. We welcome pull requests through GitHub to fix bugs, improve performance and stability, or add new features. Keep an eye out for “help-wanted” tags on issues.

We’re so thrilled to have you along with us on this journey, and we can’t wait to see where it leads. We look forward to being part of a growing community that drives OpenSearch to become software that everyone wants to innovate on and use.

To learn more, read the What is OpenSearch? FAQ.

TAGS:
Carl Meadows

Carl Meadows

Carl Meadows is Director of Product Management at AWS and is responsible for Amazon Elasticsearch Service, OpenSearch, Open Distro for Elasticsearch, and Amazon CloudSearch. Carl has been with Amazon Elasticsearch Service since before it was launched in 2015. He has a long history of working in the enterprise software and cloud services spaces. When not working, Carl enjoys making and recording music.

Jules Graybill

Jules Graybill

Jules Graybill is the Director of Search Services at AWS where he leads the development of OpenSearch and Open Distro for Elasticsearch. With an Amazon career spanning three decades, he has been involved in a variety of projects from technology to human resources. He lives in Seattle and enjoys tinkering with electronics and retro operating systems in his spare time. You can find him on Twitter at @JulesGraybill.

Kyle Davis

Kyle Davis

Kyle is the Senior Developer Advocate on the Valkey project. He has a long history with open source software development; he was a founding contributor to the OpenSearch project and most recently worked to build a community around Bottlerocket OS . When not working, Kyle enjoys 3D printing and getting his hands dirty in his Edmonton, Alberta-based home garden.

Mehul Shah

Mehul Shah

Mehul A. Shah is the GM for Search Services at AWS, including Amazon Elasticsearch Service, OpenSearch, Open Distro for Elasticsearch, and Amazon CloudSearch. His passion is leveraging the cloud to build smarter, more efficient, and easier to use data systems. He has three girls, and, therefore, he has no spare time.