Elastic Cloud (Elasticsearch Service)

I work in a gaming company where we handle a lot of microservices, observability, monitoring, and metrics. We aggregate all our logs to Elastic Search  for troubleshooting across different environments including production, staging, and dev. We use Elastic Search  to give us insights and to conduct a lot of troubleshooting.

We decided to go with Elastic Search because of the ability to aggregate everything into one portal where we have access to our entire infrastructure and the correlation about observability and traces. I have used competitors, but we are not using them in the production environment; perhaps on lower environments, but for production, we use Elastic Search.

One thing I appreciate about Elastic Search is the ability to aggregate everything into one dashboard, so I can have monitoring, logs, and traces in one portal instead of having multiple different tools to do the same.

Normally, if you were to use Prometheus, you need to know the Prometheus query language, but with Elastic Search, it gives us the ability to use normal human language for queries. It is very intelligent when it comes to querying. Unless you want to search something in depth, I find it very user-friendly.

I think hybrid search, which combines vector and text searches, is very effective because a developer or platform engineer does not need to spend time learning how to do a query. They can log in and use the standard query language to query a specific log, for example.

The initial deployment of Elastic Search was very easy for our instance because we just needed to enable some annotations for it to start getting the logs. We only needed to do a very minimal deployment on our side. The advantage we had is we had already deployed templates, so we did not need to configure each and every microservice. Once Elastic Search was there and we were able to push the annotations to our deployment, everything came alive.

I think the biggest issue we had with Elastic Search was regarding integrations with our multi-factor authentication tool. We had a challenge with the types of protocols that it allows. Sometimes you find it only supports one or two, and maybe we have a third-party tool for our MFA, so we are limited in how we can do integrations and in terms of audit. Since we are in an environment where we need to be compliant and have all our audits done, it is very hard to audit access logs for Elastic Search. I do not know if that has changed; perhaps we are still on an older version, but that has been the major issue we have experienced.

When it comes to updates for Elastic Search, we might need to push updates, for example, when they have a security patch that we need to enhance or add into our deployments. We do this in the lower environments for staging and then promote it into production. There is not much ongoing maintenance that requires any sort of downtime.

Elastic Search gives you quotas, so you are able to monitor your quotas and know when you are about to fill them up and maybe expand or tighten on your logs. Internally, we try not to have alert fatigue, so we only do important logs and queries, and we rarely have any sort of lag.

Elastic Search is very flexible when it comes to scalability. Being on the enterprise license, it is not really a big issue for us because we can increase the number of quotas we need depending on the logs we want.

For Elastic Search, we have never contacted any support. I appreciate the way they do their documentation and blogs. As a technical professional, before I reach out to support, I have to do my own troubleshooting and research; unless it is something that I cannot resolve, that is when I will probably raise a ticket. In the recent past, we have not raised any specific ticket for Elastic Search.

Before we migrated to Elastic Search, we were using the open-source tools Grafana  and Prometheus for logs, but we had to have another third-party tool to do tracing such as Jaeger, or have Sentry  to do database logs.

The initial deployment of Elastic Search was very easy for our instance because we just needed to enable some annotations for it to start getting the logs. We only needed to do a very minimal deployment on our side. The advantage we had is we had already deployed templates, so we did not need to configure each and every microservice. Once Elastic Search was there and we were able to push the annotations to our deployment, everything came alive.

The deployment of Elastic Search was done by our DevOps team, because I am part of the DevOps team. Our technical lead was mostly involved in terms of authentications and API key setup. From my side, it was easy for me to enable the annotations on the deployment and commit into the repository and push the changes to it. It was a team effort at different levels.

I would give Elastic Search probably an eight because there is always room for improvement. In IT, everything keeps evolving, and AI is here, and probably tomorrow something else will come, so they will need to elevate their game. I give it a general rating of eight, which for me means it is working perfectly, but it can always get better; there is always something to improve. My overall review rating for Elastic Search is eight out of ten.

Our use case is mainly for monitoring purposes, as we are getting the logs from our Linux machines where the applications are installed. Then we are forwarding these logs from the Linux servers to Elastic Search .

For now, we are logging the logs into the dashboard, and whenever a user wants to search on the logs, we use the platform directly on Elastic Search . I don't think we use full keywords; we directly use the user interface in the Elastic Search dashboard. Mainly, I think that should be sufficient for our users.

We don't use elastic streams for log ingestion or for structuring raw logs without agents.

We use the attack discovery feature to create alerts.

The best feature of Elastic Search that I appreciate is its monitoring capability. Whatever logs you want to forward to Elastic Search are pretty clear, and you can even edit the logs if you want some logs to delete or some logs not to appear in the monitoring dashboard, so you can clear it from there. It's pretty easy to install, easy to get handy on Elastic Search, and also easy to use it in the project. I think that's the main advantage of Elastic Search.

From a security point of view, I find Elastic Search to be quite secure, as we have a separate cluster that is well secured, and not just anyone can enter it easily.

I've noticed that the logs we are getting from the Linux servers have become automated, and in the long term, I believe Elastic Search will give promising results. When compared to Prometheus and Grafana , Elastic Search plays a main role in injecting SQL-related logs as it can inject any type of logs. It can show us any type of logs, which will be very helpful for any company or organization.

We forward the logs to our internal system that has an internal alerting system maintained by ING. The person monitoring Elastic Search, for instance, an ops guy this week or next week, will take care of the alert and try to fix it, making it quite handy to use this feature.

I think the first area for improvement is pricing, as the cluster cost for Elastic Search is too high for me. When I compare it with Prometheus or Grafana , we get very cheap dashboards with them. Elastic clusters are very costly; I understand the capabilities it has, but the price should be reduced a little bit in the market.

I also think the indexing throughput should be reduced, as using the bulk API in Elastic Search takes a lot of time and should become very fast. Additionally, observability features like search latency, indexing rate, and maybe rejected requests should be added to make the platform more reliable and accessible for everyone.

I have been using Elastic Search for close to two years in my current project.

As far as I have been using it for two years, I did not find any glitches or bugs, so I would rate it an eight or nine.

When it comes to scalability, it is scalable, but the pricing also matters, so I would rate it six or seven.

I would rate their technical support a nine because they are pretty reachable every time.

The deployment was easy for us.

We wrote some Ansible  scripts, and it took maybe two weeks, a couple of weeks.

I don't think the hybrid search that combines vectors and text searches will be in my use case.

Currently, we are not using any of the trusted GenAI experience features such as Agentic AI, RAG, or semantic search.

I recommend Elastic Search to other people because it's quite reliable when used in a project. Every project can incorporate Elastic Search because it has a lot of features. The only concern I have is pricing; other than that, the features are very good. Everyone will be able to use it easily, but you need to keep in mind that you have to train some resources because there are not many people experienced with Elastic Search. You should provide some training to them before deploying them onto the project. I would rate this review an eight overall.

I use Elastic Search , and from time to time I use it, but most of the time I am a system administrator. I deployed it more than using it. At the beginning, I was a system administrator, responsible for the deployment and maintenance of Elastic Search  clusters. For a few years now, I have started to use it more because the end users are rookie users. They need a lot of help to be able to use Elastic Search effectively. I started to be a user approximately five years ago.

Today, at least, we provide a global, unique Elastic Search cluster for the whole company, and all teams store their logs inside, their traces, and their APM  traces. Teams use Kibana to display information. We also use Prometheus exporters to collect metrics from the logs. We execute some query DSL over Elastic Search to collect metrics, which will be injected in a time series database like Prometheus. This is the main usage. We store metrics, logs, and APM  traces.

The deployment of Elastic Search is excellent. I like Elastic Search very much for that. I say regularly to the team that Elastic is elastic. It is really difficult to break. This was not the case a few years ago when I worked with Elastic Search version one and version two. Starting with version six of Elastic Search, it started to be really strong. Today, in the past, the main issue was about the data and the volume.

At the moment they integrated lifecycle policy for indices, ILM, Index Life Cycle Management. When it was created, additionally to the data stream, it started to be really easy to have all the same index volume. It is really easy to administrate and to balance data between data centers and between data nodes, and to keep the same everywhere. It is very nice. It is my favorite feature of Elastic Search. It is so easy to manage. Also, maybe because we used it for a long time, we started to be comfortable with all the setup and the node type, and how we should manage our cluster to make it resilient. I think it is really easy to maintain comparatively to some other databases.

To be honest, there is only one downside of Elastic Search that makes sense because we use a basic license, which is a free license. We do not have some features available because of the free license. Except for that, I do not have any complaint. It works perfectly. It is pretty easy to administrate and to use. I do not have complaints, to be honest, except the fact that we do not have all features available such as the APM service map or alerting.

We are not able to use a provider like Sentry , Slack, or PagerDuty. We are forced at some point to generate metrics from the logs in order to use our alerting stack in Prometheus, which works. It is an open-source project which allows us to generate alerts to Slack, PagerDuty, and some third-party tools without any license. However, it is not doable with Elastic Search in the open-source version. The alerting part is the most complicated part to manage because of the license.

From time to time we have some JVM, Java Virtual Machine issues with Elastic Search. However, it is more linked to users' requests. From time to time, people ask Elastic Search to search inside one year of logs without a nice query and without any filters. This is clearly not doable and some nodes will crash. This makes sense. However, Elastic Search is really stable when we do not have this kind of request.

Elastic Search is the perfect tool for scalability. You just need to deploy new nodes. They will be able to join and reach the cluster really easily. I appreciate it for that as well because today at VP, we use Terraform  to deploy our infrastructure. All Elastic Search nodes are managed through Terraform . If I need to extend my data node or my ingest node or whatever, I just need to deploy new nodes with the same setup, and the node will join my cluster, and it will scale horizontally really easily.

For logs management, I have not used any alternatives or something similar to Elastic Search. For APM as well, there was a plan in the past to try to migrate to Grafana , the Grafana  open-source platform for APM traces using Tempo. Tempo is a Grafana Labs project. However, we decided to keep Elastic Search for that, so we do not have any other tool or similar tool to accomplish that.

Maybe just one, it is about error tracking. We can track errors with APM inside an application, and currently we use Sentry , which is not just an error tracking platform, but also about performance management. However, we use it only for error tracking. It is more useful for developers at the beginning of a new project. Most of the time, they prefer to be connected to Sentry more than APM in order to track errors. When the project will be in production, they will be more focused on the performance than the errors. At this moment they will start to use APM, Elastic Search APM more than Sentry. We do not provide any performance indicators. Sentry is also able to manage performance metrics, but we use it only for errors and everything related to performance has been disabled.

I think the pricing of Elastic Search is really, really expensive. The main point is that we do not get any license. I tried in the past, a few times, to contact the Elastic Search team to get a quote, and it was so complicated each time to get a quote because of the volume and the number of nodes. We are a big company at VP, so we have a lot of nodes, more than one hundred. For sure it was so expensive. They tried to tell me about the enterprise mode and about the new license way to manage cost based on CPU and memory usage. It was really expensive because at this moment, we do not use any cloud services. Our Elastic Search cluster is on-premises.

Everything is self-hosted at VP tech, at VP. We do not have any limit. People using AWS  or GCP have limits because the volume of data is really expensive in cloud services and cloud platforms. Because we self-hosted everything around our services such as Elastic Search or Sentry, the idea is to let the user be able to store a lot of data and a lot of metrics. We try to train the team to have a good log level. We do not have such limitation in terms of volume. We have a really big cluster, and at the end, the price is so huge. I gave this review a rating of ten out of ten.