Elastic Cloud (Elasticsearch Service)

Elastic Search  is normally used for full-text search where users are fully depending on it for searching by name, address, and similar fields, and we need to gather the data with good latency, so we normally prefer to save it into Elastic Search .

Elastic Search helps for full-text search because we normally use it for keywords and other related terms. If there are keywords and searching requires numerical data and other elements, we prefer RDS  over Elastic Search. However, if it is regarding complete full-text search in which we cannot do any kind of indexing and it is very difficult, we prefer Elastic Search.

Elastic Search's best feature is that it is very convenient to save, plus it is schema-less, and it has very good latency and also provides us with different kinds of mapping strategies which allow us to optimize many things according to the data structure. It is a kind of non-structured and structured mix.

The benefits of using Elastic Search are mostly for two to three purposes. For logging, it is very easy to insert the logs into Elastic Search and start searching it using Kibana, and it is very easy to make visualizations over there. The second purpose is that we normally use it for views. If we have searches from the front end with a specific structure, it is very difficult to go to a different table and create the query in the database, so what we do is sync our database with Elastic Search and create a view on Elastic Search which will give us the result in milliseconds. This is how we are currently utilizing it.

Elastic Search has an annoying limitation regarding page size. It has a specific limit for queries on Elastic Search, and the default is ten thousand, and we can increase it. However, after increasing, it can slow down. Pagination in Elastic Search is very slow. If I need to parse one million records saved into Elastic Search, it becomes a nightmare because I need to do the pagination, and it is very problematic in that regard. I need to do ten thousand records and then go to the other page, and when going to the other page, it currently takes much more time than RDS . For specific cases, if we need to do full-text search and searching for one specific word returns less than ten thousand records, it works very well. However, if we go for more than ten thousand, then it creates an issue for us.

It has been almost ten years since using Elastic Search.

Elastic Search receives a stability rating of nine point five; we rely on it.

In terms of scalability, for the managed service, it is very easy, but the scalability aspect is a bit tricky. Scaling up Elastic Search cluster requires a bit of time because of sharding and replications. It takes more time since it needs to copy the data. For example, if we are working on three nodes and adding a fourth node, the synchronization process will occur in the middle, and the higher the data volume, the more time it will take. Scalability is rated around five to six.

Elastic Search's technical support receives a rating of eight.

Previously, we were using the AWS  managed cluster on the cloud, but now we have created our own. On the same cloud, we have deployed Elastic Search on our EC2  machines, so it is self-managed, not on-premises. On-premises would be if we give the solution to somebody else, then we would deploy Elastic Search on their specific cloud, but we only deployed it in our system.

I did not go into the deployment part of Elastic Search because it is a DevOps matter. I was in a senior role, so I sent the request and we received it. Normally, it does not take a lot of time if the person deploying is capable; it does not take more than two to three days.

We have about twelve specialists.

I cannot say much about the return on investment part because we normally work on a use case basis. If we find some kind of issue in our database which is currently taking time, then we need to shift to Elastic Search, and it will start giving us very good results. On the cost-saving side, rather than increasing our RDS from a less cluster to a big cluster, we can create a specific separate Elastic Search cluster, and it saves our money on our basic structure while giving us much more performance. I cannot tell you the exact part on how much was saved with the calculation, and I cannot provide the numbers, but it saves our time on the debugging side. Using it on the logs and creating visualization is very convenient for us to search the log and identify the issue as soon as possible. This saves our time, saves the customer's time, and decreases the time to respond and resolve.

Elastic Search's pricing totally depends on the server. Managed services from AWS  are used, and we have worked on a self-managed Elastic Search cluster. On the AWS side, it is very expensive because they charge based on query basis or how much data is transferred in and out, making it very expensive. That is why we moved to the self-managed option. In self-managed, it is very easy to handle. We do not think any kind of proprietary Elastic Search solution is required.

Elastic Cloud Serverless  is not being used. The GenAI experience with features like agentic AI, RAG, or semantic search is not currently being used. Kafka Streams is being used for log instigation.

Elastic Search has many pros, but the cons of it are that it is not structured, and we need to put all the things which are connected into a single index. Therefore, we cannot use it for our base structure database, but we always use it for supporting purposes.

While part of Careem, there were hundreds of thousands of customers using the solution, and now that in a startup, the clients are no more than one hundred.

Elastic Search requires maintenance. We need to keep it updated because Elastic Search normally launches new features and versions on both Kibana and Elastic Search sides. We need to keep updated ourselves, and also, we need to do maintenance on the storage side. Normally, we use Elastic Search for timelines, saving all the data from beginning to end, so normally the storage maintenance is an issue, and we have to increase the storage time to time, but it is not related to Elastic Search; it is actually related to our use case.

There is lots of support for Elastic Search in different tools like Logstash  which we normally use for integration, and there are other tools as well, but it is very easy and not a big issue for that.

The Attack Discovery feature is not being used. Big businesses cannot survive without Elastic Search because it gives us very good visibility and handles our use cases very well. If we need something reliable and trustworthy as a solution, then Elastic Search is the way to go, as it is an integral part of big solutions. The overall review rating for Elastic Search is eight point five.

Elastic Search  is being used for two main streams. The first use case is an internal analytics engine for the usage of our services, which is based on logs that are put into Elastic Search  indices to build different dashboards for key executives and developers, providing different levels of information. This is essential to provide statistics as a nonprofit organization funded by the Department of Energy and other infrastructures. The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast  and another on the West Coast , serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

The second use case is Application Performance Monitoring , where Elastic Search APM  stack is used to collect application performance metrics, primarily using Java, with a bit of Python and Node.js. Those three agents are used along with a standard infrastructure with the APM server that injects everything into Elastic Search indices for incident recovery and finding performance bottlenecks. As a nonprofit organization using an open-source license, there have been no problems with Elastic Search trying to change the license. Since no commercialized services are provided, the organization remains out of the scope of those issues and continues using open-source licenses. Recently, integration with an internal Keycloak instance was completed to provide role-based access to the Kibana application, which was a bit non-trivial but was managed successfully.

The experience regarding the relevancy of search results with Elastic Search is positive since it is used for providing search features for end-users of the Protein Data Bank. During ETL processes, information is collected from different data sources regarding proteins, including protein annotations and structures, which are transformed and loaded into the internal database. One part of that database includes Elastic Search indices. For search capabilities, full-text search is performed for end-users while keyword search is used primarily for internal needs, and no complaints have been heard about either of them.

The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast and another on the West Coast, serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

There are a couple of improvements that would definitely save a lot of headache with Elastic Search. One would be if the open-source license had multi-user access to Kibana, which exists in the paid tier license. There were also some difficult times with parallel and point-in-time interfaces, so better documentation could help, particularly more example-driven content. The provided documentation tends to have some common words but lacks real applicable examples. More specific examples, such as step-by-step guides, would be ideal. From a technical point of view, there are no significant issues recalled as Elastic Search has been absolutely awesome for this use case and covers 100% of the needs.

Elastic Search has been used for roughly five years.

Regarding stability, there are no major incidents recalled with Elastic Search. While not part of the DevOps team, nothing significant has ever exploded to affect the whole organization. If there were issues, the DevOps team was able to fix them quickly. Problems have been experienced with other services, but not with Elastic Search.

In terms of scalability, Elastic Search is good for this organization. A standard three-node setup with multiple clusters is being used for internal and public needs, resulting in six nodes per database across the data centers.

There has been no need to contact customer tech support for Elastic Search. It has been sufficient to visit conferences such as SCALE in Southern California Linux Expo, where Elastic Search has a booth to talk to their staff. The organization often relies on publicly available resources such as forums, issue trackers, and an internal knowledge base. Once, a ticket was created on GitHub  concerning a Kibana issue with Application Performance Monitoring, but that was essentially the extent of it. The main sources of support are conferences and documentation.

No alternatives similar to Elastic Search have been tried. When the discussion about the open-source license started, OpenSearch  was briefly looked into but the decision was made not to move forward because the organization felt secure in the current usage without commercialization.

Elastic Search AI, RAG, and semantic search have not been explored yet, as those opportunities for integration are just beginning. Nothing has been moved into production, so further comment cannot be provided. Standard agents from APM are being used to collect telemetry metrics and send them to the Application Performance Monitoring server, which are different from AI agents.

It is difficult to assess the current pricing of Elastic Search because the organization is in a specific niche as a nonprofit organization. On-premises instances are managed internally and a managed option had been considered, but that did not pass the board's approval. Open-source licensing has worked well, and there have been no ceilings where payment options for additional services needed to be considered. Users are quite satisfied with what is provided, and the organization is happy with what is received from Elastic Search.

The learning curve with Elastic Search was very easy. With a strong background in Java and software engineering, and having a great tutor in the organization who showed how to perform ingestion pipelines with Grok  and how to use the development environment within the stack, the process was manageable. While it might be difficult for middle-level and junior developers, having someone experienced in the organization makes it manageable to share knowledge.

Elastic Search mostly requires maintenance during upgrades. While it is running in standard mode, there have been no major incidents from memory, so it has quite low maintenance requirements.

There are no official partnerships with Elastic Search; the organization is just a user utilizing the open-source license. Overall, this review has been given a rating of 9.