Redpanda Cloud - Annual Commits (BYOC & Dedicated/Serverless)

I use Redpanda as a provider for my personal project. I create a tool to manage Redpanda, to do observability on it, to do development, to do testing, to do dashboarding, and monitoring on it. I am creating a tool to do almost most of the features as a developer, DevOps, and test person.

If I have to summarize Redpanda, it is a competitor of Kafka. The good part about Redpanda is that there is a free part, which is great. It is embedding most Kafka features. You have a number of features similar to Confluent, but for free, which is great. With Confluent, almost everything is not free. You have a dashboard for free, but it is limited. In Redpanda, you have many features, but some of them are not free, of course. However, you have more features for free than Confluent, for instance. I put Redpanda in the Kafka family, along with Confluent and Kafka.

I think Redpanda needs to increase the connector options. Kafka is well known for Kafka Connect. I think Redpanda lacks this. Redpanda really focused on many things such as contract types in Kafka, Avro, Protobuf, and JSON Schema. They manage it automatically. You do not even have to specify it in the schema registry. It can detect automatically the JSON Schema, the Protobuf, the Avro, the zip, and so on. This is great. However, I think for the connectors, they are still young, so they need to enhance the connectors with anything such as MongoDB, cloud, big data, Elasticsearch, Datadog, Splunk, MySQL, databases, SGBDR, flat file, anything. I think connectors need to be more announced. And also performance reviews. Kafka was great, but there is nothing around Kafka. You need to implement or find tools by yourself, such as Conductor. This is why I created my own tool where I manage Redpanda and eleven other brokers.

I have been using Redpanda for six months.

I have not experienced any lagging, crashing, or downtime in Redpanda.

I think the scalability of Redpanda is great because you can use Kubernetes or OpenShift because it is container-ready. If you want to have it scalable, I would recommend installing it on containers for high availability. When you have a container failing, you just kill the container and you rerun a new container with replicas. This is great.

I have not used the technical support that Redpanda provides because I use the free version. I develop alone on this. This was one of my previous employment and skills before. I continue by my side. I do not have any subscription and I never had one. I had a subscription on IBM MQ a lot. I have subscriptions on TIBCO EMS and Rendezvous, subscriptions on SonicMQ, and so on, but never on Redpanda. I only use free features, so I do everything by myself. Unfortunately for the support management, I have no knowledge.

The initial deployment of Redpanda is fairly as easy as installing ActiveMQ or RabbitMQ, which is very easy. In complexity, I think the most complex is IBM MQ, really. Mostly, of course, when you do clustering, it is very difficult and high availability, and so on. This one is very easy. I put eight on ten. The premium version has an analysis tool for when you have a license, saying this has to be enhanced, this is too tiny, you need to put it bigger, this is too big, you need to put it smaller for messages, for topics. It explains your ISR is not well configured and so on. So it goes further but it is cheaper than Confluent because Confluent is so expensive. I think it is a good competitor but still young, so it needs more experience to be at the level of Confluent.

Redpanda requires maintenance on my end. Of course, you need to check that you do not explode the size of the disk space. You need to check that the performance is still good with the ISR repartition. It is doing that, but you still need to check that everything is okay on the hardware level of your machine because it consumes resources, of course. I think this needs alerting for the disk space, for the rotation of the partitions in the topics, and so on, just as Kafka does. You need to check that it is still based on the ISR, of course. You need to check that there is no latency. You need to do performance tests sometimes to check that everything is still fast, mostly when you use streams.

I also used the Docker version. I used the installer first and then I migrated to Docker.

It is a difficult question to give a score to Redpanda overall. Unfortunately, you cannot give a score objectively. I will try to be objective. It is difficult to give a score based on what you need. If you want reliability on messages, I would recommend IBM MQ. If you need something that is compared to Kafka for streams, for performance, for failover over the regions, then I would say you can use Redpanda. I would give Redpanda an overall rating of seven out of ten.

We used Redpanda as our main message hub for processing different messages asynchronously and with regards to the consumer's throughput.

The biggest main feature that Redpanda offers is the same Kafka API, so you could always use the already-built libraries and the big environment that is around Apache Kafka as a message broker. There are a ton of libraries in every possible language, and then you could seamlessly plug Redpanda instead of Apache Kafka and use that instead.

Through a lot of benchmarks and whatnot, we found out that on a larger scale, when the latency of the messages is important, not talking about throughput, the latency is important, Redpanda offers a way better latency profile than Apache Kafka. So, we used to have Apache Kafka and then we switched to Redpanda.

When you claim that you are Kafka-compatible, and in our product we use a ton of libraries built for Kafka, and then in some really, really specific niche cases, they had different ideas on how to do something, and that would make the library kind of useless or not work as it's intended. Because in some cases, especially when it was about the way a leader broker is selected and things related to the consensus of your cluster of Redpanda, it was way different than the behavior of Kafka.

One of the reasons that we migrated was back when Zookeeper was a different thing that you had to manage and configure to have a Kafka cluster, but Redpanda was plug-and-play, ready to go. You could have one node or ten nodes and then the consensus and leadership, everything was obvious and ready to go from the start. So deployment-wise, it is a really good product.

The most important thing that Redpanda could do better is, first of all, define the way that they are different from Kafka, and then try to fork some libraries so that we could use those specific libraries built for Redpanda rather than us using the Kafka ones. I would assume that because the differences are very minor but very niche, they could have essentially the same library and the same amount of code that is already there for Kafka and then change it a bit so that you could use the Redpanda client instead of the Kafka client.

In my career, in total, I was at a company for three and a half years and it was a main stack for message brokers, so we can say three and a half years. But me personally interacting with it was one and a half years into the company, so two years.

We had one incident that we concluded was a case of memory leak. Back then there was a similar issue in their GitHub regarding this specific bug that we encountered. After a while, we just patched it with the new version and everything was fine. I do not know about systems with ten thousand microservices and how they would react in that situation, but in our system where the latency and the throughput were way more important with less amount of things integrated with Redpanda, it was fine. We only had this major issue.

It is pretty scalable. We have used the scalability features, and we have added different nodes to the already up and running cluster. As I mentioned earlier about the consensus and the leadership thing that is kind of different from Kafka, we encountered some specific bugs related to that when we wanted to add a new node to the cluster. Overall, I would rate them a seven out of ten. I do not say that is the most scalable message broker there is, because I guess Kafka is way better in scalability, but it is not terrible either.

I do not have technical support. However, they have a Slack community and Slack channel, and we used to communicate with them a lot. In so many cases, the developers themselves answered us because we were heavily benchmarking what they said and what they claimed that they were. At one point, about three years ago, we contacted them and we had this Google Meets meeting that we discussed what we found and what they developed. Not the technical support as in the usual way, but the community and the development support was great.

I have used Apache Kafka as an alternative.

It was very easy. That is one of the reasons that we migrated. Back when Zookeeper was a different thing that you had to manage and configure to have a Kafka cluster, but Redpanda was plug-and-play, ready to go. You could have one node or ten nodes and then the consensus and leadership, everything was obvious and ready to go from the start.

Since we were self-hosting the open-source version, we had to maintain it, but I do not think that the maintenance was a burden. I was more into developing things and integrating things with Redpanda. The maintenance team of infrastructure was a whole separate team, but as far as I know, it was not a big hassle.

Unfortunately, when we were using Redpanda, we were self-hosting the open-source version, so I do not really know about the pricing.

I have used Apache Kafka as an alternative.

We used to use Apache Kafka and then we switched to Redpanda. I kind of appreciate the way they are going with their documentation more than Apache Kafka. However, Apache Kafka, because it is used in a lot of environments with a thousand or ten thousand pods and ten thousand moving parts and all of those things, in those cases I guess Apache Kafka would still surpass Redpanda. But in case of low-latency systems, Redpanda is way better. This review has an overall rating of eight point five.

For my current field, I process data, but for Redpanda specifically, it is more for data that occurs live. For example, in my project, I use Redpanda to get MTA subway feeds live so that I can process them later with another tool such as Spark for partitioning and saving it into S3 buckets for storage.

I do want to mention that there was a part of Redpanda where you can configure and create containers with Docker. That is one of the valuable features. The sub part of it is that you can configure the retention size to a specific number of megabytes, and that way, when you go to the producer or the consumer part, you can see how much messages are being retained. Even if you configure it, you can have the messages there for a week or so, even if you don't have it running.

I would say Redpanda is pretty stable overall. However, in terms of the topics and the retention, it could definitely be improved. For me, I was trying to get a lot of data, but I would only get one day's worth of data, even though I asked how I could increase the retention size. I was told it would be seven days worth, but in practice, it would just be one day's worth. The way it cleans up the old data meant that I would basically have to run the consumer job 24/7. I would say the biggest scalability improvement could be the retention.

I was looking at other features on the UI, but I am not familiar with them. I don't know if there is documentation for them, but there are other tabs on the UI that I am not really familiar with. Maybe because it is hosted locally, I am not really sure. It would be helpful if there was some sort of documentation or an instruction set for how to integrate other software to make the workflow much easier.

I want to say over two years, but this year specifically, I have gained way more insight compared to the previous years.

I have not experienced any stability issues.

I would say Redpanda is pretty stable. In terms of the topics and the retention, scalability could definitely be improved. For me, I was trying to get a lot of data, but I would only get one day's worth of data, even though I asked how I could increase the retention size. I was told it would be seven days worth, but in practice, it would just be one day's worth. The way it cleans up the old data meant that I would basically have to run the consumer job 24/7. I would say the biggest scalability improvement could be the retention.

I have not really thought about it because I am not familiar with other similar products.

It was a little bit difficult because when I was creating the containers through Docker, part of it was trying to figure out whether I wanted the whole workflow to run locally. For me, that consumes a lot of power. So I had a virtual machine run Redpanda exclusively, but the configuration for that, for the local host and all that, was a little bit confusing. Maybe that is just the virtual machine aspect and not really a Redpanda thing. But other than that, I think setting it up was not too complicated. It is just the virtual machine portion of it.

Locally, I would say it was around an hour. Not too much.

In terms of maintenance, not necessarily. I would say it was pretty straightforward. The only nuance to it was making sure the JSON data was compiling properly, and then setting up the retention size.

That is kind of the issue right now. I don't have a company verified email. I am mostly using it for personal purposes with my personal email for everything. I just never got the hang of how to create a company email.

My overall review rating for Redpanda is 7 out of 10.