Keeping clients of OpenSearch and Elasticsearch compatible with open source
September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. Visit the website to learn more.
The OpenSearch project is a long-term investment in a secure, high-quality, Apache-2.0 licensed search and analytics suite with a rich roadmap of innovative functionality. OpenSearch aims to provide wire compatibility with open source distributions of Elasticsearch 7.10.2, the software from which it was derived. This makes it easy for developers to migrate their applications to OpenSearch: Existing clients, connectors, and low- and high-level APIs will just work. Our experience at AWS is that developers find it painful to update their already-deployed applications to use new version of server software, so backward compatibility for clients and APIs weighs heavily in our designs.
Many developers who use Elasticsearch and OpenSearch in their applications also make use of the open source client libraries maintained by Elastic, which provide convenient high-level interfaces for several popular programming languages. Over the past few weeks, Elastic added new logic to several of these clients that rejects connections to OpenSearch clusters or to clusters running open source distributions of Elasticsearch 7, even those provided by Elastic themselves. While the client libraries remain open source, they now only let applications connect to Elastic’s commercial offerings.
The most broadly adopted open source projects generally emphasize flexibility, inclusion, and avoidance of lock-in. Developers often rely on this expectation of interoperability when choosing the dependencies they’re going to use, so changes like this are disruptive. And indeed, we’ve heard from developers that these changes are creating challenges for them. This includes anyone who depends on these clients and runs open source versions of Elasticsearch, Open Distro for Elasticsearch, OpenSearch, and customers of Amazon Elasticsearch Service, in addition to developers and open source project maintainers who want to support the broadest range of search backends in their applications and projects.
To give these users a clear path forward, the OpenSearch project will add a set of new open source clients that make it easy to connect applications to any OpenSearch or Elasticsearch cluster. These clients will be derived from the last compatible versions of corresponding Elastic-maintained clients before product checks were added. In the spirit of openness and interoperability, we will make reasonable efforts to maintain compatibility with all Elasticsearch distributions, even those produced by Elastic. These clients will let developers continue running their current version of OpenSearch or Elasticsearch with minimal changes to their application code. The new clients will offer the same APIs and functionality they use today. The list of clients we will create forks of includes:
In the meantime, for anyone using one of the above distributions of OpenSearch or Elasticsearch, we do not recommend updating to the latest version of any Elastic-maintained clients, as this may cause applications to break. Anyone who has already updated their clients and is experiencing issues can resolve these issues by using any of the client versions outlined in the OpenSearch documentation.
Great open source software is built together, with a diverse community of contributors. Open source development gives stakeholders the opportunity to shape a project’s direction by contributing to it directly, and by serving as leaders. As part of this initiative, we are seeking community comaintainers for each of these new client libraries. Maintainers will have a leadership position for the library and will play a key role in setting its direction and priorities. If you are interested in participating at any level in the development and roadmap of these client libraries, visit the OpenSearch website for details.
Within the next few weeks, we anticipate opening repositories for each new client library, in the OpenSearch GitHub organization. We will then work quickly to publish the clients to all the corresponding package managers and repositories. Together, we will ensure that developers have easy access to well-supported client libraries for open source Elasticsearch, Open Distro for Elasticsearch, and OpenSearch.