Honeycomb Enterprise

I am part of the performance engineering practice, and I lead the performance engineering practice at my current employer. We use Honeycomb Enterprise  for tracing, which is application performance management in short. Our client has several APM  tools such as Datadog , and in addition to Datadog , we only have the monitoring capacity of the counters. We do not have the agent-level monitoring which Honeycomb Enterprise  is providing, where we can see the traces for each call being made by the software to trace where it is spending the time. For that gap, they have Honeycomb Enterprise in addition to Datadog. We use Honeycomb Enterprise for the same purpose, as Honeycomb hooks into our applications and tells us the traces where the request is spending the time.

We have Datadog here and often we get restrictions related to cardinality on Datadog because of their billing systems. They have limitations of cardinality, and that is the impact. That is how we can compare the impact.

Another thing I want to add here is that the team here had tried to use Honeycomb Enterprise earlier for tracing, but they faced issues. They could not get proper tracing with Honeycomb Enterprise at that time. That is what I have been given as feedback.

My main focus is on the tracing part. We have a microservices architecture with multiple microservices, and we want to see how when the request flows across multiple microservices, where the time gets spent.

We mostly look at the time the requests spend. Honeycomb Enterprise would capture a trace and span, and from there, we look at the time, the milliseconds or seconds that get spent at a particular request. That is what we look at and what we are interested in.

I did not get a chance to utilize the BubbleUp feature, and I think this is the most famous feature for SREs. We have not been able to utilize BubbleUp.

Dynatrace  has this PurePath technology where they have the ability to give end-to-end tracing, which is a very robust feature. AppDynamics also has certain good features such as we could deploy that on our database systems and it would give an entire picture of our database, showing which application is creating how much load. From a pros perspective, Honeycomb Enterprise could be a better candidate with high cardinality; when there are too many unique values, Honeycomb Enterprise could be more beneficial there.

I have used better tools, I would say. I would not say that I prefer Honeycomb Enterprise as much. I have used Dynatrace , and I found it more comprehensive, and AppDynamics and other tools. These tools can also provide good information, but I find other tools better.

Most of the products, I would say, such as Dynatrace or AppDynamics or New Relic , are targeting this microservices market. I think Honeycomb Enterprise can have something very dedicated for microservices because there is an explosion in the migration from monolithic to microservices. If Honeycomb Enterprise can create a stable solution which is easy to use and which gives additional value and helps for faster debugging with microservices, they can certainly gain market share from others.

Tracing is already there. I just wish that these tools are a bit less cryptic. These tools sometimes get quite cryptic for new users. The less cryptic they can be made, that can help these tools. Another thing is that for microservices, when you have multiple microservices installed, that is also required. There are tools where you install on a single microservice, but then these microservices interact with multiple microservices. That kind of picture, I have seen that in AppDynamics; they do give a picture showing that a particular request which arrived here had interaction with these other third-party services or microservices and databases. That is what we need. That is what performance engineers and SREs need to see for each request, where it spent the entire time; how many other services or databases it interacted with and what took more or less time, and if there is a sequence, it should highlight that also. Was it parallel or if, for instance, a call to service A and then a call was made to a database, or a call to service A and a database were in parallel, that kind of information.

I have used Honeycomb Enterprise for around a year.

Another thing I want to add here is that the team here had tried to use Honeycomb Enterprise earlier for tracing, but they faced issues. They could not get proper tracing with Honeycomb Enterprise at that time. That is what I have been given as feedback. This was also one of the reasons the team is less interested in using Honeycomb Enterprise. Most recently, as far as I know, they started using an open-source tool now, Jaeger. They also brought in Jaeger because when I was using Honeycomb Enterprise, I got this feedback that they could not use Honeycomb Enterprise meaningfully in the team, although they had this license and everything. What I know is that it was crashing with the application. These tools, as they instrument into the application, they can sometimes lead to crashes. I had seen this with AppDynamics also, but that was for a different client. There, we could see AppDynamics did provide a fix for that. The team did follow up with AppDynamics. They had a wider setup of AppDynamics in the company, so they could not just get rid of that. A similar feedback I also got from the team that when they were trying to use Honeycomb Enterprise for tracing, it was causing issues.

As I said, there have been issues with Honeycomb Enterprise. The team had reported to me that at one point in time when they were trying to use Honeycomb Enterprise, they faced stability issues. It was interacting with the application and causing some crashes in the application.

I would rate the stability of Honeycomb Enterprise around five.

Frankly speaking, we did not use it at such a large scale here, so I cannot comment on that. The Datadog we use here is quite scalable. That is being used for at least eight thousand hosts. Given the cloud version and all, the scalability depends on our provider, the service provider who gave us the Honeycomb Enterprise license. That is something which is beyond my scope.

The setup sounds a bit complex. The overall setup at our client's end is really very complex because they have high security limitations, which makes it a bit difficult to use any of the tools.

When other organizations are evaluating their APM  tools, I could name Honeycomb Enterprise as one of the APM tools, another addition. There are not that many APM tools, with the top ones being Dynatrace, AppDynamics, New Relic  and a few others. I heard about Honeycomb Enterprise in my current project only; I did not hear about Honeycomb Enterprise before. Given the new technologies emerging, if Honeycomb Enterprise can provide better traceability and monitoring, they have a chance.

Another thing is that for microservices, when you have multiple microservices installed, that is also required. There are tools where you install on a single microservice, but then these microservices interact with multiple microservices. That kind of picture, I have seen that in AppDynamics; they do give a picture showing that a particular request which arrived here had interaction with these other third-party services or microservices and databases. That is what we need. My overall rating for Honeycomb Enterprise is five out of ten.

My main use case for Honeycomb Enterprise  is alerting and full stack tracing. I use Honeycomb Enterprise  for tracking our traces from the front end all the way to our back end in Kotlin, and through Kafka and into Elastic. This helps us to spot any bottlenecks and see where errors occurred in our data processing pipeline.

We primarily use Honeycomb Enterprise to power our alerting, so we get alerts based on throughput, based on spikes, anomalies, and any error rates in each service. The best features Honeycomb Enterprise offers are alerting, which is mainly what we use it for.

Visualization is fine for me, although our dashboards are a bit cluttered. Primarily, the alerts are the most useful thing. The querying is good, although sometimes a bit messy. I would appreciate a better way to store my queries, perhaps with some auto-fill functionality. The alerts are the best feature.

Honeycomb Enterprise has positively impacted our organization by providing live alerts. We have been able to get alerts when something may pop up and see any issues in the system. We get alerts into Slack, and they work great. We see a lot of metrics go through into Slack, and they are really useful for keeping our team focused on only seeing one place to see alerts.

Honeycomb Enterprise can be improved by having a cleaner dashboard and a nicer way to search between insights.

I have been using Honeycomb Enterprise for one and a half years.

I do not have the metrics for a return on investment, but it has been very useful.

Having those alerts and metrics in Slack helps with our SLAs. If we ever have downtime in a service, we can get back to it straight away. I rate Honeycomb Enterprise a seven out of ten because I feel a lot of the journeys could be made cleaner. Some of the dashboards and the insights could be a bit more well-refined. For example, Grafana  has a lot better visualization tools, but it needs to be a bit more involved.

Honeycomb Enterprise is deployed in our organization in a public cloud and is hosted by Honeycomb, who is the provider.

My experience with pricing, setup cost, and licensing is that the number of samples would be good. The number of events, really, so we can pay less. I do not have the metrics for a return on investment, but it has been very useful. It is quite expensive, but I think we get value out of it. My overall rating for Honeycomb Enterprise is seven out of ten.

We were building a product for one of the biggest wealth management platforms in the world, an American wealth management platform. For them, it is really important for the product to be reliable and for them to set up KPIs, especially for vendors like us who worked for them.

The debugging process usually involved Splunk Cloud or Honeycomb Enterprise  traces. Whenever I was looking at an issue, I probably went through the traces because it was a microservice architecture. Sometimes it really helped to understand the call chain. For example, if there were 10 microservices calling each other in some sort of order, being able to visualize that and look through that was pretty useful.

I would say all of these three are pretty good features of Honeycomb Enterprise .

Honeycomb Enterprise is super useful if you think before this paradigm shift, when it was really important for humans to be able to see things visualized. For me to better understand in this very complex microservice architecture what's going on, Honeycomb Enterprise really helped with that through its good UI. However, the reason it's only five is because it's lagging behind in terms of AI-compatible features.

The major thing that's missing from Honeycomb Enterprise is AI compatibility. As far as I know, it's not really a text-based or code-based tool. It's more of a UI right now, which before this paradigm shift where everyone is using AI agents to work, was pretty useful. However, after this paradigm shift, I think it's really important for a tool like this to be AI-friendly. Honeycomb Enterprise is really lagging behind in this area, and I don't know how they could manage that.

For us it was sometimes pretty slow using Honeycomb Enterprise. I don't know if that can be improved. Although we might use the paid solution or self-host it somewhere because of privacy concerns. Maybe that's not Honeycomb Enterprise's fault. However, the main thing is that I think everything should very hard aim for the direction of being AI compatible because every engineer, or most engineers now use AI to code. If something is not easy to work with AI agents, that will stay in the past.

I used Honeycomb Enterprise at my previous job for around one to one and a half years.

Honeycomb Enterprise is stable.

It's very scalable since we used it for a really big organization and it worked.

If it's a big company, I understand using Honeycomb Enterprise. If it's a small company, I would suggest using some sort of open-source solution or something that is code-based and is easier to use with AI.

Although Grit  is a tool code code migration and management of technical debt for large chunks of work, we reviewed Grit  from the use case of assisting in faster remediation of vulnerable libraries. We examined 3 areas and how we could use the synergy of Grit.io along with Snyk .io that helps overcome Snyk 's limitations:

1. Deep scanning and reachability analysis

2. Management of auto-generated Pull Requests (PRs)

3. Reduction of false positives

I'm connected and had interactions with the founder Mr. Morgante Pell, while I designed a comprehensive synergistic solution, and I wrote a 35+ page technical paper on this topic.

Large organizations with hundreds of development teams and tens of thousands of code repositories face challenges in efficiently identifying and remediating potential vulnerabilities within third-party libraries across numerous projects. Manual scanning and updating is time-consuming, error-prone, and can lead to delays in addressing security risks.

While Grit.io is not primarily a vulnerability scanner, its pattern-matching and code transformation capabilities can be adapted for mass identification and remediation of vulnerable libraries.

For each of these 3 areas listed above, we examined how Grit.io's unique features can complement Snyk.io's capabilities, resulting in a more robust and efficient security scanning process. We realize This synergistic approach addresses the limitations of relying solely on Snyk.io, resulting in improved code security and reduced risk of overlooking critical vulnerabilities.

The limitations of security scanning tools like Snyk.io represent real challenges faced by development teams on a daily basis. These limitations can lead to:

- Missed vulnerabilities in complex code structures

- Overwhelming numbers of auto-generated PRs, causing developer fatigue

- High rates of false positives, leading to wasted time and resources

We considered implementing Grit into our pipelines to address these specific scenarios for code security, though Grit isn't a security tool:

- Custom Rules and Pattern Creation

- Remediation Pattern Creation

- Automated Code Updates

- Custom Pattern Recognition

- Pull Request Generation

- and others

1. Grit.io's flexibility allows for custom rules and patterns to identify vulnerable libraries, extending its use beyond traditional refactoring tasks.

2. Automated pull requests streamline the remediation process, facilitating efficient mass updates across multiple repositories.

3. While not a replacement for dedicated security tools, Grit.io can be a valuable addition to a large organization's security toolkit for vulnerability identification and remediation.

4. The approach offers significant benefits in terms of efficiency, consistency, and proactive security management, particularly valuable for organizations with large, distributed development teams.

I asked very specific questions to Mr. Pell about consideration of code security scenarios in pattern design and rules, specifically that tuned with OWASP Top 10. I believe addition of code security focus can be a value-add, though the way Grit architecture is designed and how it works, it is and may not become an alternative choice of code security solutions. Rather, it must be treated as a powerful supplementary tool that augments the existing code security solutions (such as Snyk or Checkmarx) in a DevSecOps  or Secure DevOps environment.

Anyone interested in learning more on this front or have queries, can get in touch with me for a consulting.

Our internal comprehensive evaluation of Grit spans over 6 months to a year since our client organization considered Grit under the Accelerator program of promising AI startups back in Sep 2023. Different phases of the implementation have been conducted by various development architects spanning several scenarios. Our scenario was very specific to how Grit's AI-powered capabilities could be leveraged on code security remediations for a large tech ecosystem.

The solution is mainly used for stack observability. It observes service behavior or any kind of failure that may be happening. The tool is also related to research. My company is working more on this, but I have been working on my SLOs and defining SLOs for the last seven months.

The solution's most valuable features are the queries for the OpenTelemetry events and all the tracing. The solution is very easy to use, and the dashboard is very intuitive.

We faced some OpenTelemetry metrics lost between the communication from the service and the Honeycomb.io. I can't say if this is a Honeycomb.io issue or if there are some limitations in OpenTelemetry.

Alerts are very helpful in Honeycomb.io, but we don't usually merge because we can compare queries with queries for making alerts. We can make alerts based on static numbers, which may block us from building alerts that could be generic enough or could be serviced.

I have been using Honeycomb.io for two years.

We’ve never had any issues with the solution’s stability.

Honeycomb.io is a scalable solution. The service is very resilient and can handle a lot of data. The quantity of data Honeycomb.io can parse and use to create charts is really good. More than 100 users are using the solution in our tech team.

I rate the solution’s scalability an eight or nine out of ten.

The solution's initial setup is easy. I think the hardest part is to understand OpenTelemetry in general.

We set up Honeycomb.io on all the services so that we can have all the set traces of the communication between all the services inside the company. This helps us understand where it could be failing, which in turn helps with failures and observability.

When we are not thinking about one specific failure but a major one, we can create queries for statistics views like the P99 or P95 behaviors. That's very helpful. I would recommend the solution to other users because it is very helpful.

Overall, I rate the solution a nine out of ten.