Elastic Cloud (Elasticsearch Service)

The main use cases are for logging, centralized logging system, and security purposes. We also use it for application monitoring and APM  to monitor all the applications that run in our environment.

Applications developed by some of our users are monitored using APM , which is one of our primary implementations. For security purposes, we centralize logging for all 6,000 servers using Elastic Search . With more than 12,000 servers in our infrastructure, we need to track which server requires attention and receive alerts. For example, if we need to update all servers, some may be missed, but the system will trigger an alert to notify us. Monitoring and logging are the main functions we use in our current systems.

We are using Elastic Search  for log ingestion only.

I chose Elastic Search because it has high search capabilities and setting up the cluster and maintaining it is very easy. Due to this, I found it very user-friendly. High availability and shards allocation are significant advantages that led us to shift to Elastic Search.

I particularly appreciate the sharding concepts because data has high availability. The semantic search feature and the new logsDB feature are valuable additions. These are things I appreciate most about the platform.

Semantic search is a very advanced feature that has proven useful for our data in current systems. I am working with Aadhaar, which is a Unique Identification Authentication firm. When we search for name-related terms, the semantic search provides relevant results. I have also implemented semantic features with hospital data, and it has been very useful for multiple cases.

Elastic Search Hybrid Search is an advanced feature that functions as a future vector database. Vectors are the main component of the database. In current systems, it shows only similar data, but with a vector database, we can store all types of data using vectors. Everything in the future will revolve around vectors. All systems are moving from CPUs to GPUs. This is very useful because comparing vector databases will be a more efficient way to store and retrieve data compared to traditional methods.

Pricing is very high compared to other solutions, but given the features they provide, the pricing is acceptable. The licensing part is also decent compared to other features. I have no issues with this because the features they provide are excellent and position us for next-level future capabilities.

Many banks are moving to Elastic Search, and many identification systems are adopting it because the search capability is significantly higher compared to other solutions, and data retrieval is also very efficient. Many industries are transitioning from old solutions like Splunk to Elastic Search. Banking sectors and healthcare sectors are leading this adoption. Many applications use Elastic Search as their backend, such as Zama. Industries are thinking about and adopting Elastic Search technology because of the features it provides.

There are several areas that need improvement. First, while storing data, there are many mapping issues and mapping conflicts that cause Elastic Search to reject the data. We have to develop solutions or significantly change our processes to address mapping conflicts. This is one of the issues that needs to be fixed.

Second, building semantic search requires significant setup and configuration work. If Elastic Search could provide a one-shot, easy-to-use semantic search implementation, many more users would adopt it. Currently, only a few users are using semantic search, but if they brought it with one-shot ease of use, many people could use it easily and create alerts.

Third, Elastic Search Vector Database needs more attention in the market. We need to bring more features about the vector database to make it easier to set up and use. The use cases also need to be brought to market. Additionally, building dashboards in Kibana is challenging. Compared to Grafana , Kibana has very few features and chart options. We need to enhance Kibana to allow very customized dashboards to be built. Kibana needs significant enhancement in this area.

I have been using Elastic Search for five years.

Elastic Search is stable and reliable until you build the cluster for one terabyte. If data reaches one terabyte, it functions well. However, if data exceeds that or reaches a bottleneck, it becomes unstable. If data is at eighty hundred gigabytes or seven hundred gigabytes, which represents seventy to seventy-five percent of the built cluster capacity, it is very stable and reliable. Search latency is very low compared to other solutions like ClickHouse . Stability and reliability are completely dependent on the data volume.

From the scaling perspective, horizontal scaling by adding extra nodes works well when data increases. We can easily add nodes into the cluster and scale horizontally. Vertical scaling is also straightforward where we can increase the size. We can add new nodes and new components very easily.

I have raised ticket sizes with them many times. I feel very supported by their customer service. For P1 tickets, they provide very immediate quick responses and join calls to support and troubleshoot the issue accordingly. They provide solutions very efficiently. Their service is very good.

I have used Splunk and Dynatrace  previously.

I have worked with ClickHouse , and there were many issues with indexing while storing data. The approach is different with ClickHouse. I have also worked with Splunk, and it functioned adequately. However, when storing large setups or large amounts of data, Elastic Search capability is superior and is really useful for the end user.

I believe the initial setup for this solution is complex for new members. However, if you are technically strong and understand how Elastic Search systems work, it is very easy. With five years of experience, I have set up many clusters for banking sectors and healthcare sectors. I have built fourteen clusters in production environments with large-scale systems exceeding five terabytes. This will be typical for those who have technical knowledge and can build easily. Those starting without experience can use Elastic Cloud, which offers very easy one-click deployment. They can deploy an Elastic Search cluster with single clicks. Those with technical knowledge can build the cluster themselves, but those without experience can use Elastic Cloud. This is not an issue.

Correlation alerts is a feature I did not get the opportunity to work on. I have only theoretical knowledge but not practical knowledge.

We can use agentless approaches with a script in addition to agent-based approaches. We are building both agentless and agent-based solutions. Both are good. Agent-based approaches for fetching data work well. Both are functioning well.

Discovery is a feature we are using, and it works well. Attack is a feature I did not get the opportunity to try.

Elastic Search is very user-friendly, and we can easily integrate it with third-party models and other AWS  S3  buckets. It is very user-friendly for integrating with other third-party tools.

My overall review rating for this solution is ten out of ten.