I have used Apache Kafka on Confluent Cloud for one of my projects with regard to log monitoring. My main use case for Apache Kafka on Confluent Cloud in that project was mainly streaming of the logs. I wanted to capture logs coming from various interconnected systems into a unified place, so Confluent helped me to streamline all those logs into one place, and then I was consuming those logs that were produced.

Having all my logs unified helped our team a lot because the main challenge we were trying to solve was that in the current scenario we were working on, there was no place where we could view the logs in one place like Grafana or anything similar. It was not available, and for this scenario, we had to use multiple systems in order to check the logs, which could be databases, different applications, and logs for various other APIs. So it was not unified in one place. Now we unified all those logs by producing it to Apache Kafka on Confluent Cloud and then we were consuming it.

It was very much easier because I know Kafka and using Confluent made it much simpler. It was much more easy to understand, grasp, and very well structured with regard to even the JSON response and everything.

I am using Apache Kafka on Confluent Cloud with the Confluent servers itself; we had taken a subscription to Confluent, so it would be the private cloud.

In terms of the development it took us to set this whole thing up using Confluent, we were able to do it at a quicker rate. If we went with the ideal vanilla Kafka, it would require much more manual effort, but here it was easier because of the user interface and the experience, which was mainly very much drag and drop, so we could easily get it done faster.

We did initially have Grafana, but the only problem with Grafana was it was limited to certain applications, and ours was not among them. Because of that, we thought of shifting to a unified log monitoring system and started off with having vanilla Kafka installed on our servers. But we found Apache Kafka on Confluent Cloud much more convenient regarding how we would set it up, so to get things done faster, we shifted to Confluent.

The best features Apache Kafka on Confluent Cloud offers would be the connection with various external systems through various languages such as Python and C#. That gave us an edge to use this tool, and we were able to connect to it.

The multi-language support helped my workflow because we did use two kinds of languages, mainly .NET or C# and Python. So we were able to get logs because we had these out-of-the-box connectors available in Confluent. We just had to read the documents, understand how to configure it, how to send the logs, and we were through by just adding a few lines of code in our applications in the respective languages.

Apache Kafka on Confluent Cloud positively impacted my organization because this was just one of our projects we had utilized, while there are many more I was not involved in, so I cannot really speak on them. But this helped a lot with regard to log monitoring and many other streaming use cases where we were able to get all the data in one place and stream it at a quicker rate, and it was much more simpler compared to using plain vanilla Kafka. So Confluent made that difference for us.

I think Apache Kafka on Confluent Cloud can be improved by probably working more around Confluent or the tool. In my opinion, it should utilize the response structures in a better way or be able to detect if there is any variable or if there is any data structure that is mismatched, as it would be easier than us manually having to put in the exact name in order for it to match the response.

Regarding additional improvements, I would say probably around error handling, where when we encounter errors specific to our response structures and everything, or the tables or anything of that nature, it would be better if we were prompted with better error handling mechanisms.

I do not think there are any other improvements Apache Kafka on Confluent Cloud needs, aside from error handling and response structures.

I have been working in my current field for four or more years.

Apache Kafka on Confluent Cloud is stable.

Regarding scalability, I think it depends on the subscriptions and the way we set it up on the cloud. According to me, it is quite scalable in terms of all the data it can handle and stream, so I would say it's quite scalable.

I did interact with the customer support team when I was checking on my credits issued and the cost of my credits utilized. During that time, I had an interaction through email, and it went very well. I was getting prompt responses, and it was nicely handled regarding the support.

Positive

We did initially have Grafana, but the only problem with Grafana was it was limited to certain applications, and ours was not among them. Because of that, we thought of shifting to a unified log monitoring system and started off with having vanilla Kafka installed on our servers. But we found Apache Kafka on Confluent Cloud much more convenient regarding how we would set it up, so to get things done faster, we shifted to Confluent.

In terms of the development it took us to set this whole thing up using Confluent, we were able to accomplish it at a quicker rate. If we went with the ideal vanilla Kafka, it would require much more manual effort, but here it was easier because of the user interface and the experience, which was mainly very much drag and drop, so we could easily get it done faster.

We did not really look at any other options. We were, in fact, given Apache Kafka on Confluent Cloud as the top solution when we were doing our solution designing, so that is the reason we directly started our testing and trials on Confluent Cloud.

I do not have metrics to share regarding the return on investment, so I cannot really give an insight on that.

My experience with pricing, setup cost, and licensing went well. When I did my testing around it or research, I used the free credits. However, I did not realize that I was crossing the credits until I received an email regarding billing, so I had to email the team to let them know that I was not aware of that. I thought Confluent would stop me when I crossed the credits, but it did not, and then I got charged. So, because of that, there was back and forth with the team regarding the cost, but overall, I feel it would have been easier to handle if there were a mechanism that let us know when the credits were being fully utilized or when the limits were getting crossed.

We did not really look at any other options. We were, in fact, given Apache Kafka on Confluent Cloud as the top solution when we were doing our solution designing, so that is the reason we directly started our testing and trials on Confluent Cloud.

My advice to others looking into using Apache Kafka on Confluent Cloud is that it is easier and has a low learning curve. If there is any use case regarding streaming, I would suggest starting off or definitely trying out Apache Kafka on Confluent Cloud and getting the application maybe set up on it, if not evaluating how it returns you the returns. This is a good way to start off because of how it is easier to set up; it saves time, allowing us to utilize the time for other business logic or other things we would need to do. I would rate this solution a 9 out of 10.

Positive

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have, and other features. KSQL also helped us there.

When order is more important, we rely on Apache Kafka on Confluent Cloud.

The benefits that I have seen from having a real-time architecture include better velocity for developers. That is the main one. Instead of developing many of those capabilities in each team, we can rely on Apache Kafka on Confluent Cloud to provide those functionalities we want, and the teams can focus on their own business instead of providing all sorts of APIs and dependencies to other domains, allowing everyone to run faster.

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have. KSQL also helped us there.

I think what I would improve about the solution is the cost, mostly. From my standpoint, it's the cost. From an engineering perspective, it works really well.

There's always room for improvement. One more point is sometimes it's more UI-related issues. Some of the more high-end features are more complicated to execute. But overall, it's a good product.

I have been using Apache Kafka on Confluent Cloud for around a year, maybe two.

When it comes to assessing the impact of the automated scaling features, we don't measure it, but it's part of our technology stack selection criteria - it's pretty much a must today.

We don't want to increase the headcount in our DBA team. They are the ones managing all our databases, queues, and data sources. So for us, having a very thin layer of management is critical, and we sit with other compute. That's very important for us because headcount is the most expensive part.

We looked at other products, specifically other Kafka providers. We have Apache Kafka and AWS. We looked at self-hosting it, but we wanted Apache Kafka on Confluent Cloud.

Neutral

We were looking for specific use cases. We compared different Kafka solutions, not necessarily competitors. We have a message bus already. We wanted the log capability, mostly.

The setup was easy enough. We got a lot of support from people at Confluent and AWS as well.

Regarding ROI in any capacity, whether it's savings from employees or cloud, the ROI was very significant. Although, specifically with Apache Kafka on Confluent Cloud, it was a bit more challenging to increase adoption because it's very expensive. So we had to pick and choose where we implemented to make sure that ROI is positive.

I don't remember the exact number because it's been a while since we did a pricing talk, but it was expensive.

They charge per topic and other resources. Because we are very cost sensitive, we want to approve it and make sure people don't just use it unnecessarily.

I would give Apache Kafka on Confluent Cloud a rating of seven out of ten.

For somebody who's shopping around, looking in this space to decide what to purchase, Apache Kafka on Confluent Cloud is a market leader. It's almost the first choice.

Going with AWS Apache was also very compelling to us because it's very quick to enable stuff in AWS and try it. I would start with those, but first understand if this is actually what you need. There are other much cheaper solutions that serve other use cases, and sometimes people can mix those and just pick the wrong product.

Overall, I would rate Apache Kafka on Confluent Cloud a nine out of ten.

Positive

Neutral