Elastic Cloud (Elasticsearch Service)

My use case has evolved over time with Elastic Search . Initially, we started with it as a searching solution. Before Elastic Search , our primary source of truth was SQL databases, the traditional RDBMS . We thought about taking the data from the traditional RDBMS  because they were not able to cater to the scale that we wanted to achieve, so we migrated the data from MySQL , keeping it as the primary source of truth, but for the searching mechanism and wildcard searches, we migrated to Elastic Search.

My experience with the relevancy of search results in Elastic Search includes both traditional keywords and full-text search. In the supply chain industry, with millions of orders and customers such as CMA CGM, Maersk, or Kuehne+Nagel, filtering out those orders was essential, using a shipment number, transportation order number, or an origin or destination number. In the gaming industry at FDJ United, full-text searches make more sense to understand gaming intent. For example, when a user searches for 'I really want to play action games', we break down that full-text query, use custom text analyzers, and derive the intent behind the user's query in combination with a vector database alongside Elastic Search.

My assessment of the effectiveness of hybrid search, combining vector and text searches, shows that Elastic Search is remarkable for text-based searches. I have explored other solutions, but none can beat Elastic Search in that area. When I combine hybrid searches with vector databases, they store the mathematical representation of the data. For instance, to find the top 10 closest proximity based on a query, the vector database uses cosine similarity on the available data and suggests the top 10 results while Elastic Search can keep the metadata, enabling quick access to the entire database based on derived intent.

I have utilized trusted GenAI experiences related to semantic search and text-based search in my current project using Elastic Search. My go-to solution for text-based searches will always be Elastic Search, but for semantic search, I am trying to build a solution that emphasizes system-level understanding agents. For example, if a new engineer queries the agent for a system explanation, it scans all the relevant data and provides a comprehensive analysis of the service, contextualizing inputs to reduce hallucination, controlled temperatures for the LLM model, and reducing nucleus sampling. As for knowledge preservation, I use a vector database to store significant outputs generated by the LLM, depending on user preferences regarding the gravity of the analyses performed.

The best features of Elastic Search that I appreciate include its capability for eventual consistent systems where you do not need hard consistency, and it scales very smoothly. For wildcard searches and regex patterns, it really scales massively. It offers ILM, indexation lifecycle management, which allows you to enable a search for a span of six months for the data fed into the system while moving the rest to a new cluster. The structure of the inverted index document facilitates its core features, and I find how Elastic Search understands, indexes, and creates mappings for your data to be remarkable.

While Elastic Search is a good product, I see areas for improvement, particularly regarding the misconception that any amount of data can simply be dumped into Elastic Search. When creating an index, careful consideration of data massaging is essential. Elastic Search stores mappings for various data types, which must remain below a certain threshold to maintain functionality. Users need to throttle the number of fields for searching to avoid overloading the system and ensure that the design of the document is efficient for the Elastic Search index. Additionally, I suggest utilizing ILM periodically throughout the year to manage data shuffling between clusters, preventing hotspots in the distribution of requests across nodes.

I have been using Elastic Search for more than six years.

In terms of stability, I would rate it eight out of ten regarding downtime, bugs, and glitches.

For scalability, I assign it a ten out of ten.

I would rate Elastic Search's technical support as nine out of ten.

Before Elastic Search, our primary source of truth was SQL databases, the traditional RDBMS.

Estimating the return on investment from Elastic Search is nuanced; however, I can share that initially, search times from traditional RDBMS were around two to three seconds, and with Elastic Search, we reduced that to 50 milliseconds, indicating a significant improvement.

Assessing the complexity of deploying Elastic Search, I have a gray area because a separate DevOps team handles that aspect, but from my experience writing code and utilizing its features, I find it not complex at all.

Estimating the return on investment from Elastic Search is nuanced; however, I can share that initially, search times from traditional RDBMS were around two to three seconds, and with Elastic Search, we reduced that to 50 milliseconds, indicating a significant improvement.

On the subject of pricing, Elastic Search is very cost-efficient. You can host it on-premises, which would incur zero cost, or take it as a SaaS-based service, where the expenses remain minimal.

When comparing Elastic Search to other vendors and products, I have recently explored Algolia , which is also a fully managed service. Elastic Search offers a choice between hosting on-premises or as a fully managed service, which has been beneficial compared to other solutions.

In my company's relationship with the vendor, I have always worked in product-based companies using Elastic Search, often as part of solutions from companies such as Manhattan Associates and in the gaming sector. For B2B industries, they sold to large clients such as Maersk and CMA CGM while my current company, Agoda, operates in the B2C space.

Elastic Search does require some maintenance, especially when considering features such as ILM if you want to enjoy its capabilities. Maintenance tasks depend on the established data pipeline and may introduce some friction.

Currently, we are not using Elastic streams for log ingestion; previously, we utilized the ELK and EFK stacks with Logstash  for log ingestion and Kibana for visualization. I also observe a trend where companies migrate to Grafana Loki  instead of ELK.

Regarding integration aspects, Elastic Search has exposed REST APIs for all its services, making it easy to integrate with third-party models or endpoints regardless of the underlying infrastructure, as any modern development language can interact with these REST services.

I have not used the attack discovery feature.

My deployment of Elastic Search is on-premises.

At Agoda, we handle over 1.2 billion searches daily, facilitated by Elastic Search.

While I have been at my current company for four months, I am still getting to know my colleagues; however, I know there is a dedicated team focused on Elastic Search. This team exposes a service that acts as an intermediary for communication between Elastic Search and other services.

In my department, there are more than 100 people, whereas the overall organization consists of thousands, exceeding 10,000.

I would rate this review overall as a nine out of ten.

Elastic Cloud (Elasticsearch Service)  is a wonderful solution for seamless implementation and maintaining its health. It is much more reliable in the cloud than the on-premises issues that occur very frequently on-premises. However, Elastic does not cover the whole world, and in my region, the Middle East, there are very few hosting places for Elastic Cloud (Elasticsearch Service) . It is good news that Elastic recently invested in hosting Elastic Cloud (Elasticsearch Service) in Saudi Arabia, set to launch in March, which I anticipate will lead to more customers adopting Elastic Cloud (Elasticsearch Service) in the very near future.

The only way to visualize data in Elastic, whether it is on-premises or in the cloud, is using Kibana. Kibana's cloud version is not different from the on-premises version, but Elastic Cloud (Elasticsearch Service) is usually more up-to-date, as Elastic maintains and consistently updates Elastic Cloud (Elasticsearch Service) to the latest version, while on-premises versions may lag behind.

I assess the machine learning capabilities of Elastic Cloud (Elasticsearch Service) as truly exceptional, although it is the least used and least understood among many customers. There are quick features that customers can benefit from, such as anomaly detection, but they can also add their own models, which some customers perceive as complex because they do not understand machine learning models and need to have data scientists on their teams to utilize that capability. If a customer uses machine learning in Elastic Cloud (Elasticsearch Service) heavily, they will find that it is very fast to get results compared to using other tools.

Machine learning might be expensive for customers. Customers take advantage of Elastic being open source, but machine learning is not available in the open source version. If a customer is using the open source version without paying licenses to Elastic, they will not enjoy the machine learning features. That is why machine learning does not have the same popularity as Kibana and the other components in Elastic, because only those who pay for Elastic can experience it.

Regarding additional features I would appreciate seeing in the next release of Elastic Cloud (Elasticsearch Service), Elastic acquired Gena AI, and I would appreciate seeing more AI models embedded in the upcoming new versions of Elastic Cloud (Elasticsearch Service). This is what I will be waiting for.

I would rate overall Elastic technical support a seven. It is very noticeable that they are good and responsive, but they heavily collect a lot of logs from customers before resolving issues, which makes the support ticket take longer than expected.

Some of my customers utilize Elastic Cloud (Elasticsearch Service), especially in the private sector, but most of the government sector do not use it.

Elastic Cloud (Elasticsearch Service) performs well. There are two types of Elastic Cloud (Elasticsearch Service): hosted Elastic Cloud (Elasticsearch Service) and Serverless  Elastic Cloud (Elasticsearch Service). Serverless  is more expensive compared to hosted Elastic Cloud (Elasticsearch Service), and controlling your bills in serverless sometimes becomes unpredictable, more often than in hosted Elastic Cloud (Elasticsearch Service). Hosted Elastic Cloud (Elasticsearch Service) is not adaptive; it does not rely on data rates, and you will know your spending from day one until the end of the year because unless you change the size of Elastic Cloud (Elasticsearch Service). As long as the size of Elastic Cloud (Elasticsearch Service) is constant, your bill is constant. With serverless, the bill changes frequently based on the influx of the data rate.

I assess Elastic Cloud (Elasticsearch Service)'s ability to handle diverse data sources such as logs and metrics as very good. Elastic managed to unify their data collection through Elastic Agent, the new version of Beats, allowing you to collect various types of data with the same agent. Elastic Cloud (Elasticsearch Service) is performing well in this area, although some data still needs to be ingested by Logstash , but Elastic Agent keeps improving over time.

My overall rating for this product is nine.

I have completed two different Elastic Search  implementations, and in both cases, the goal was to speed up very slow Postgres databases. As a platform PM, I am typically responsible for user management and company management. These areas are quite heavy depending on how many users, customers, or companies exist. Before Elastic Search, when we relied solely on Postgres, there were significant delays to user list pages and company list pages. In the other company, there was a lot of data displayed for particular list pages for admins. We combined Postgres with Elastic Search to speed this up, and it certainly does speed it up. We have used it throughout my current job and previous job.

From the customer side, Elastic Search is super fast and very efficient, delivering results quickly. We recently tuned a series of compliance results in the CMS where we would specify that certain results should come up higher by adding keywords and other factors. However, the results were not as good as when we restarted and used Elastic Search out-of-the-box search results. We actually got better results that were more logical.

The most significant issue I find with Elastic Search is that it gets out of sync, and this has happened in both cases where I have implemented it. When Elastic Search gets out of sync, for instance, if I create a company or user, it gets created in Postgres and then sometimes there is a delay for it to appear in Elastic Search. This could be a 15-minute delay depending on how it was implemented. If other significant processes are running on the platform where you are touching a lot of records, such as a million records, that will take a hit on Elastic Search. We have seen differences of 800 records between Postgres and Elastic Search. Proactive tools that would find and adjust any mismatches would be beneficial.

Occasionally, Elastic Search has failed, and when that happens, search results do not come up at all. This has been a rare occurrence, and I am not certain Elastic Search is entirely to blame, as it could have been platform storage or other factors. For the most part, the most common problem is the out-of-sync issue.

I have used Elastic Search  since 2019 at the last two companies where I have worked.

I would say Elastic Search is pretty scalable. We have had good results.

Earlier, in the 2019 and 2020 range, we were having a lot of trouble with syncing, and we tried to see if consulting was available. At the time, we could not find what we needed from the knowledge bases, and we could not really get support. There was not a technical support option at that time. That may have changed. Currently, I do not think we have gone to Elastic Search to ask for any significant help.

Google Appliance was a search engine that Google had for a while, and we used that and were pretty happy with it, but then they deprecated it and it is no longer available. After that, I do not remember what we used, something else.

Although I am not an engineer, it seemed easy to medium to set up. It was not complicated.

For the initial rollout, I would say it was maybe two or three people for the initial implementation, and then I have a team of 40 or 50 engineers with somebody always working on updates and other tasks.

We have not yet used anything in combination with Elastic Search, but that is on the roadmap.

I would say Elastic Search's relevancy is okay, and if I were to give it a score, I would give it a B. Elastic Search works best out of the box as much as possible. When you start to overtune or put in other factors that will increase the priority of specific results you want to come up, it gets really complicated and then you do not necessarily get the best results.

Elastic Search works best when used out of the box without excessive tuning. My overall review rating for Elastic Search is seven out of ten.

I use Elastic Search  for fast search of products in our database. With Elastic Search , we use full-text search with keywords and different rules from the Elastic Search documentation. I do not have cases when a search request is four sentences long. I typically use three, four, or five words for searches.

I think the best feature of Elastic Search is the speed. It is very fast and comfortable to use in requests with transpositions rather than full requests. It has a smart engine inside.

In Elastic Search, the improvements I would like to see require many resources.

I have used Elastic Search for two or three years, though I do not remember exactly which it is.

Maintenance of Elastic Search is easy because we do not have problems. I would rate the stability of Elastic Search at an eight.

I would rate the scalability of Elastic Search at an eight.

I did not have a situation where I needed to ask something in technical support for Elastic Search.

I used a different solution before using Elastic Search. It was Sphinx.

I do not know if the deployment was easy or complex, and it is also not my responsibility.

I do not know how it was purchased as it is our DevOps responsibility. I know that it is in AWS , but I do not know the details of how it is deployed there.

I do not know about features such as Agentic AI, RAG, or Semantic Search in Elastic Search. I did not know that there are AI search features available.

I would recommend Elastic Search to other people who want to have fast search in their applications. It is comfortable, it is fast, and it is very interesting to work with it. I gave this product a rating of eight out of ten.

I work in a gaming company where we handle a lot of microservices, observability, monitoring, and metrics. We aggregate all our logs to Elastic Search  for troubleshooting across different environments including production, staging, and dev. We use Elastic Search  to give us insights and to conduct a lot of troubleshooting.

We decided to go with Elastic Search because of the ability to aggregate everything into one portal where we have access to our entire infrastructure and the correlation about observability and traces. I have used competitors, but we are not using them in the production environment; perhaps on lower environments, but for production, we use Elastic Search.

One thing I appreciate about Elastic Search is the ability to aggregate everything into one dashboard, so I can have monitoring, logs, and traces in one portal instead of having multiple different tools to do the same.

Normally, if you were to use Prometheus, you need to know the Prometheus query language, but with Elastic Search, it gives us the ability to use normal human language for queries. It is very intelligent when it comes to querying. Unless you want to search something in depth, I find it very user-friendly.

I think hybrid search, which combines vector and text searches, is very effective because a developer or platform engineer does not need to spend time learning how to do a query. They can log in and use the standard query language to query a specific log, for example.

The initial deployment of Elastic Search was very easy for our instance because we just needed to enable some annotations for it to start getting the logs. We only needed to do a very minimal deployment on our side. The advantage we had is we had already deployed templates, so we did not need to configure each and every microservice. Once Elastic Search was there and we were able to push the annotations to our deployment, everything came alive.

I think the biggest issue we had with Elastic Search was regarding integrations with our multi-factor authentication tool. We had a challenge with the types of protocols that it allows. Sometimes you find it only supports one or two, and maybe we have a third-party tool for our MFA, so we are limited in how we can do integrations and in terms of audit. Since we are in an environment where we need to be compliant and have all our audits done, it is very hard to audit access logs for Elastic Search. I do not know if that has changed; perhaps we are still on an older version, but that has been the major issue we have experienced.

When it comes to updates for Elastic Search, we might need to push updates, for example, when they have a security patch that we need to enhance or add into our deployments. We do this in the lower environments for staging and then promote it into production. There is not much ongoing maintenance that requires any sort of downtime.

Elastic Search gives you quotas, so you are able to monitor your quotas and know when you are about to fill them up and maybe expand or tighten on your logs. Internally, we try not to have alert fatigue, so we only do important logs and queries, and we rarely have any sort of lag.

Elastic Search is very flexible when it comes to scalability. Being on the enterprise license, it is not really a big issue for us because we can increase the number of quotas we need depending on the logs we want.

For Elastic Search, we have never contacted any support. I appreciate the way they do their documentation and blogs. As a technical professional, before I reach out to support, I have to do my own troubleshooting and research; unless it is something that I cannot resolve, that is when I will probably raise a ticket. In the recent past, we have not raised any specific ticket for Elastic Search.

Before we migrated to Elastic Search, we were using the open-source tools Grafana  and Prometheus for logs, but we had to have another third-party tool to do tracing such as Jaeger, or have Sentry  to do database logs.

The initial deployment of Elastic Search was very easy for our instance because we just needed to enable some annotations for it to start getting the logs. We only needed to do a very minimal deployment on our side. The advantage we had is we had already deployed templates, so we did not need to configure each and every microservice. Once Elastic Search was there and we were able to push the annotations to our deployment, everything came alive.

The deployment of Elastic Search was done by our DevOps team, because I am part of the DevOps team. Our technical lead was mostly involved in terms of authentications and API key setup. From my side, it was easy for me to enable the annotations on the deployment and commit into the repository and push the changes to it. It was a team effort at different levels.

I would give Elastic Search probably an eight because there is always room for improvement. In IT, everything keeps evolving, and AI is here, and probably tomorrow something else will come, so they will need to elevate their game. I give it a general rating of eight, which for me means it is working perfectly, but it can always get better; there is always something to improve. My overall review rating for Elastic Search is eight out of ten.