Elastic Cloud (Elasticsearch Service)

I am familiar with Elastic Search  to a certain extent as I have used it in my development life. I thought someone wanted feedback about it, specifically how I have used it in my career, so I agreed to share that information.

I started using Elastic Search  after becoming acquainted with it when I accessed the AWS  environment for the first time during the COVID period. We tried to establish a vertex and edge graph database schema, and I was hired to get that schema up and running while dealing with millions of records related to car spare parts. Due to a signed clause, I cannot go into too much detail. The challenge was with the indexes slowing down, which prompted a move to GraphDB because it provides faster access time. I had to deal with a lot of data cleansing and created many pipelines, first pushing records into Elastic Search through a bulk insert. I also looked up data using Kibana as the front end to leverage queries for pulling up that data.

Once GraphDB was in place, I was required to develop a service for asynchronous processing and order confirmation, where one copy would be stored in a database and the other would be pushed into Elastic Search for further lookup, eliminating the need for direct queries to the RDS 

I have never reached out to Elastic Search's technical support team.

Elastic Search, being a vector database, quickly indexes data, allowing for searches based on text and data directly, which I found fascinating. My dev lead mentioned that it uses C++ to pick up these indexes and pulls up records incredibly fast, in nanoseconds, keeping me interested in how things are becoming faster over time and diversifying away from traditional relational database systems.

Regarding scalability, I consider both vertical and horizontal scalability in theory. I have not experienced sharding but find it interesting as a use case with Elastic Search. I see significant potential for vertical scalability, which can accommodate more data and offer substantial improvement.

Your question about what I dislike about Elastic Search is quite pointed, and I prefer to look at it as something for improvement, such as provisioning options other than Kibana. A standalone install that is operating system agnostic could run on Mac, Linux, or Windows by just providing a URL, username, and password to access the schema for queries. This would benefit many people who may not have access to Kibana, especially those who, the workplace evolution has shown, may not know what Kibana is if they lack tool access. It is crucial to have executable information to understand a product deeply. If Kibana is not a viable option for everyone due to hosting constraints, a standalone installer could connect directly to Elastic Search, with documentation readily available online to guide those needing desktop access.

I have been using this solution for two years overall and have had good exposure to it with all CRUD operations I have been performing with it.

I have used Elastic Search for log lookups with ELK and never encountered any crashes or downtime while it was hosted in the cloud. While occasionally one or two queries may take longer due to network lags, these issues are more infrastructure-related since I have never faced any problems with Elastic Search's stability, which generally retrieves information instantly.

Regarding scalability, I consider both vertical and horizontal scalability in theory. I have not experienced sharding but find it interesting as a use case with Elastic Search. I see significant potential for vertical scalability, which can accommodate more data and offer substantial improvement.

When discussing initial deployment, the specific attribute of interest is the overall initial installation when starting to roll out the product. The deployment was a struggle as I faced challenges with bash commands and understanding how to run things on my system. Looking up tutorials on YouTube was tricky, and cross-referencing with documentation posed difficulties as some people customize setups to their needs. Setting up MySQL  is straightforward, while with Elastic Search, I had to run bash commands for proper service execution. I faced some hurdles getting CRUD queries to work correctly. I resorted to Docker  as an alternative, which diverged from standard practices of creating a local database service. An ideal setup would include a setup executable for Windows that would greatly facilitate immediate access and CRUD operation starts.

In my case, the system was already running by the time I started, as the custom DevOps team managed the deployment, and I was only tasked with connecting via Kibana and issuing bulk insert commands.

I have not checked Elastic Search's pricing thoroughly, so I do not know how a company would perceive it. From what I see, small companies might consider the cost, with starting pricing for a single node instance at $16 a month for serverless and hosted options, though at least one or two connected clusters would be necessary for viable solutions. Companies might see this lower end pricing as suitable, but for startups, reaching up to $2,000 could appear steep, depending on their aggressive usage approach.

I faced a situation where our graph database work halted due to technical difficulties with the Neptune product, as some CRUD operations were not carried out. Product specialists suggested that the business case did not fit the graph database's requirements and recommended Elastic Search instead for a better use case. I was involved in a data structure related to car spare parts needing to facilitate purchases by linking parts to various car makes across catalogs, ultimately attempting to shift from relational databases due to overwhelming data generation that slowed down indexed lookups. Elastic Search significantly helped in confirming order data lookup, but costs for clusters in further development led to work being stalled.

A preliminary architect consultation or proof of concept on cluster purposes would aid in establishing understanding for further development on Elastic Search, which is becoming increasingly costly in the cloud due to demand. A structured understanding of costs tied to usage metrics would greatly assist in planning before commitments, as delays in our POC adversely affected our progress. Documentation should also encompass potential use cases and scenarios to better assist developers during implementations across programming languages to ensure seamless integration.

Regarding alternatives, I have worked with various database products, including Azure  technologies where I worked with NoSQL storage tables similar to AWS  DynamoDB, which are schema-less with varying attributes per record. These use partition key and row key for accessing information, fragmenting what we associate with traditional RDS . Additionally, I worked with Axelor CRM  from a French company, alongside MySQL  and Oracle. My first company used MS SQL , and I have discussed my use case involving AWS Neptune graph database and Elastic Search, which encompasses all I have worked with so far.

For an overall rating of Elastic Search, I would score it at a solid 8 out of 10.

Its speed has facilitated my understanding of logical operators and streamlined query issuance. I would love to grasp the inner workings of sharding with distributed schema implications. Based on what I have experienced thus far, I find it a significant improvement, but once I better understand sharding and its performance effects, I would likely adjust my score.

My experience with the relevancy of search results using Elastic Search indicates that issuing a full query yields a finite number of results, while partial text searches can return irrelevant information. Mastering query issuance with Elastic Search is a valuable skill that develops over time. I prefer a structured JSON approach, utilizing properly sequenced clauses, which allow drilling down to a limited set of records that directly relate to the search context.

On hybrid search effectiveness, I think that AI is progressively offering more concise information. Providing more relevant keywords allows Elastic Search to generate results faster than other databases, such as RDS. The ability to engage with text directly simplifies understanding records and has a significant impact on AI functionality in rendering accurate results based on user needs.

We use Elastic Search  for a research application based on paper study, and the primary usage is for indexing the data and then functioning in a similar way to an e-commerce search bar.

For us, what I can notice is the ability of adding weights to each field of the data, which is very useful because sometimes the user searches the data not just by the title, but by specific keywords, and being able to add weight to the fields in order to show that information to the final user is very useful. Also, the panel for showing graphs about the data and how the users are interacting with it is pretty useful.

The difference in performance of Elastic Search  is outstanding; if we compare a traditional database or service for search and index products or, in this case, papers, the difference is outstanding. That is the case when you want to filter the data; the primary advantage will be performance for sure.

Again, the primary improvement will be performance, and the interactivity we can have with the data is very flexible; it adapts to the needs of the user very easily.

I cannot see any issues at this point; the panel is great. The way to customize and configure the panel and the search is great; it is really visual. Documentation is great as well.

The initial configuration could be easier; at first, the learning curve is a little high, and over time, it becomes easier. For me, the initial configuration might be improved.

I have around three years of experience.

Stability has not been an issue; it is working perfectly in that aspect.

Scalability has not been an issue for now.

In the case scenario when we need to face support, support was really useful, and they answered the questions in a good period of time.

Cassandra  was one we were evaluating, but we preferred Elastic Search because the documentation was way better and the community was bigger. It is easier to find answers when we face a problem, and that is why we chose Elastic Search.

At first, we faced several issues related to some versioning and allowing indexing the database because part of our information is in a traditional SQL database, and we were using the IDs from the index for the records in Elastic Search. We created a little ETL for that, and handling that process was tricky and harder at first. That was the biggest challenge we faced when starting to set up Elastic Search.

I would say that first, contact support for the initial setup; I think it will make the process easier. Then start, for example, with how to send and retrieve the data in the documentation; I think that is the best thing they can do.

For that one, my field, the PO and the technical leader is the one that handles the bills about Elastic Search.

I am on the side of implementing it, so in terms of cost-efficient or the price of using it in the cloud, that is not something I am really involved with; I am more on the dev-ops side.

It was great; the developer experience is great when integrating either the frontend or the backend side. Nothing so complex could not come.

For implementing Elastic Search, I would say good documentation, and it is really easy to use. We have an example of almost every functionality that is inside of Elastic Search framework, so that is helpful. I would provide a rating of ten for this product, and I say a ten; it is really good.