Redpanda Cloud - Annual Commits (BYOC & Dedicated/Serverless)

We use Redpanda mainly for three purposes. The first is the standard Kafka-like data ingestion pipeline where you get an event and want to distribute it into multiple applications, not just one. The second, which is our main use case, involves capturing video recording data. When you consider how many video cameras exist even in a single city or metro, the volume becomes substantial. We capture what happened at each second, but we do not directly push it into the database. Instead, we pass it through Redpanda to ensure that everything follows a certain shape. Sometimes the data is corrupted, incomplete, or missing information. For example, an amount cannot be arbitrary; it has to be a specific number. We perform all those cleanups in our code while flowing the data through Redpanda with schemas in place. The third use case is backpressure management. If you want water to do the dishes but receive a fire hose instead, we ensure that you get the right amount of quantity. We keep this very simple, and after a certain point, it becomes a day-to-day operation.

Redpanda can handle real-time financial data streams and support aggregations. For example, with stock data where each tick represents a price point per second—such as Apple stock being 252.3, then 258, then 247—you can create means, medians, and averages. All these calculations work perfectly as long as your functions are pure. You get all the data without making additional API calls or database calls, flow it through Redpanda or any Kafka instance, and perform all kinds of calculations very quickly. The database is generally the slower component. Every single time we found it was almost always the database that gets slower or the way we normalize data causing issues. Redpanda never gave us any problems. I remember RabbitMQ used to give us a lot of problems, but all those problems are gone with Redpanda.

The video example I shared involves webcams. If you consider webcams as IoT devices, we use them that way and it just works. It is not real-time in the strictest sense, but it is almost real-time, which is good enough for our case.

One thing I really appreciate about Redpanda is that it is simple and available to host yourself. You do not have to pay money upfront to Confluent or have API contracts. Unless you want very high availability, you can just host it on your own server by running the Docker container and Redpanda works. This is very good. The other valuable feature is that migrating from one version to another has almost always been a very smooth experience. With Kafka, you have to manage a thousand things. These two are big benefits in Redpanda. The third feature is that the UI is slick. I would not say it is one of the best UIs in the world, but it is much better than what Kafka or Confluent comes up with out of the box.

One area for improvement is providing more examples. For instance, Redpanda could be more useful as a sink where you get the data and can directly push to S3. While this is possible through the API, there are better and faster ways to do it. You can make a million API calls and accomplish the task in one and a half hours, but the same thing can be done in ten minutes through other methods. These faster approaches are not documented in obvious places. You have to find information scattered across various blogs. Redpanda should collect all the good blogs and best practices and put them in their documentation. This is more about knowledge management and making it easy for users to understand the product for complex use cases. For simple use cases, it is straightforward. We all use the basic pipe functionality. However, providing more examples would be useful. For example, integration with AWS and the AWS ecosystem would be cool.

We have been using Redpanda for at least the last four years.

Stability has been pretty good. I would rate it around eight or nine. Once in a while we restarted, but overall it is pretty stable.

On scalability, we always scaled vertically. When 16 GB of RAM was not enough, we went to 32 GB RAM, then 64 GB RAM. We never scaled horizontally by adding one machine, then two machines, then three machines, and so forth. We just have a three-machine setup because if one machine fails, there is a backup, but it is not to distribute the load. In a way, it just works. I would say the scalability is around eight, nine, or ten. However, I do not know how well it scales horizontally. They obviously make big claims, but I never tried it because we just scale vertically.

We previously evaluated Confluent and Kafka itself. I would not dare to go with Kafka because it requires way more orchestration than required. Confluent was good, and Redpanda was also good. However, Redpanda was an order of magnitude cheaper than Confluent, so we went with Redpanda.

The initial setup for Redpanda is very smooth. Compared to Kafka, which feels like it requires a lot of knobs and pieces, Redpanda feels like a thirty-minute setup. This was actually one of the reasons why we wanted something we could just use rather than master first and then use. We started using Redpanda and then mastered it along the way because our use case is to flow it through the pipes, and that is done beautifully with Redpanda.

We use Docker Swarm and deployed Redpanda on-premises.

The pricing for Redpanda is very good. I would rate it around nine.

We evaluated a couple of alternatives and liked three of them: Confluent, Redpanda, and others. However, Redpanda was simple and fast, so we went with Redpanda and it just works. Under the hood, it is just Kafka. They say that since it is written in C++ and Rust as a high-performance Kafka, it is fast. All the programming libraries we have are written for Kafka, and our Kafka is Redpanda. Since Redpanda provides compatibility with Kafka SDKs and APIs, we do not use Redpanda-specific SDKs or APIs. I cannot tell you how fast or slow Redpanda is compared to the original Kafka because the only Kafka instance I use is Redpanda. Redpanda is fast enough, but I cannot say that the original Kafka is slow because I never used the original Kafka.

In my opinion, I am not sure about features for decision-making processes because I do not even know what kind of features are there for that. We just use Redpanda as a pipe. My overall rating for Redpanda is eight and a half out of ten.

Redpanda  serves two primary purposes for our organization. First, we use it as a drop-in replacement for Kafka. Second, we utilize it for streaming analytics.

We do not use Redpanda  for IoT data streaming, though it has been quoted as suitable for that use case. IoT data streaming is actually a superset of our use cases. Recently, we have started using it for AI analytics as well.

The best features of Redpanda include its lightweight nature and low resource consumption. Because we migrated from Kafka to Redpanda, this is the main selling point for our organization.

Aside from its lightweight design, Redpanda is essentially a clone of Kafka with all the good features of Kafka. The only difference is in resource requirements. Kafka needs too many resources while Redpanda is a very good, lightweight, and very fast database.

We use Kafka API compatibility with Redpanda, which helps significantly with data migration. There is no need to migrate data because it works out of the box. As a streaming solution, we never migrate the data. We simply have a clean state and replace Kafka with Redpanda. In my case, there was no need to migrate.

Redpanda’s low latency data processing is excellent. We do not experience too much latency, and it works very well for our product architecture. It is exceeding our estimations. While designing the product, we design it for a certain throughput, and Redpanda always works at that throughput.

The metrics I use to evaluate effectiveness in any real-time applications are throughput. Every application has its own throughput requirements, and so far, Redpanda is good at achieving the targeted throughput for our applications.

In Redpanda, the areas that have room for improvement are in the clustering part. Setting up clustering initially is very easy. However, if you are removing a node and attaching another node, significant improvement is required. It is not as expected, and we have had a tough time adding and removing nodes.

Other solutions lack clustering support, but Redpanda has good clustering support, though it needs further improvements for smoother cluster operations.

Aside from clustering, Redpanda is good, but it is worth noting that it is built on top of a special architecture. It works directly at the Linux kernel level. Because of that, it needs better modern hardware with a better CPU, not just a normal CPU. A server-grade CPU is required. It also needs modern memory, at least DDR5. It is not good with very old computer memory. Disk type matters as well. It is very good at NVMe  SSDs but not good with old spindle hard drives. If a customer has modern hardware, it works very well. If a customer has legacy hardware, it will not work as expected.

I have been using Redpanda for more than three years.

Regarding stability, Redpanda is good. I can rate it ten out of ten for stability.

Redpanda scales very well. I would rate it ten out of ten for scalability.

My thoughts on support for Redpanda are that it is nine out of ten.

When I compare Redpanda with other solutions, such as Confluent  Kafka and Amazon, I find that overall Redpanda is good compared to any other option. Kafka is good, but it is too resource intensive. Redpanda is good and requires fewer resources.

I find that the deployment of Redpanda is easy. It takes just less than an hour to deploy Redpanda. Operations are a bit tough, but deployment is easy.

In terms of pricing, Redpanda is free. We do not have to pay anything. It is not open source, but it is free.

I would recommend this product to other users. Overall, I would rate Redpanda as very good and would give it a ten out of ten rating.

I have worked with Redpanda  for the past two to three months. Mainly in the tech industry or software industry, there's a huge rise of streaming data.

Redpanda  serves as a very reliable and fast message broker, which lets you build applications asynchronously. The major use case is for my project specifically, we're using it for a monitoring system that we're building.

I haven't worked on enterprise-grade applications. It's a project for academic research for the college.

The industry standard for this kind of platform is Kafka. Confluent  Kafka has acquired it. Kafka is an open-source platform built by Apache. Confluent  is the commercial version of it.

The major improvement of Redpanda over Kafka is firstly, good documentation. Redpanda's documentation is very easily understandable, and they have a lot of examples. In addition to that, most of the setups include using another technology called Docker , which I am very familiar with.

Setting up technologies using Docker  is very convenient to me, and Redpanda has provided many templates for that. Redpanda has its own built-in metrics exporter, making it easier to monitor and check performance.

What makes Redpanda superior is its performance since it's written in C++. C++ is pretty much the standard for high-performance applications.

Recently, for the documentation, they've built their own AI chatbot, which is focused on giving you answers based on their documentation. While using that, I did not find it to be very good. Maybe due to the fact that it's a first prototype and was very recently released. However, from a product perspective, I do not have any problems.

I have used Redpanda for the past two to three months.

I am definitely satisfied with the stability provided by Redpanda.

In the free version, it's still working with containers itself. They've provided the template for one and three nodes for horizontal scaling.

For our project, we've set up five containers. The template is ready in the Dockerfile they provided. You just add the existing nodes and give them the respective configurations.

I have not escalated any questions to technical support.

The industry standard for this kind of platform is Kafka. Confluent Kafka has acquired it. Kafka is an open-source platform built by Apache. Confluent is the commercial version of it. I have worked with Kafka, so I pretty much know how it works. I found Redpanda's documentation and setup to be more straightforward.

Once I learned about this platform, I read about it a bit and then I looked at how they set it up on their systems. It's a simple straightforward Dockerfile. If you have Docker installed on your system, you can just spin it up quickly without any issues.

Redpanda is actually a commercial platform, but they do provide free versions as well. I've been working only with the free versions.

Anyone finishing their bachelor's degree in computer science engineering would have had some hands-on experience with Kafka and understand how it works. Shifting from Kafka to Redpanda would be very simple for them.

I'd rate the solution ten out of ten.