We use Redpanda mainly for three purposes. The first is the standard Kafka-like data ingestion pipeline where you get an event and want to distribute it into multiple applications, not just one. The second, which is our main use case, involves capturing video recording data. When you consider how many video cameras exist even in a single city or metro, the volume becomes substantial. We capture what happened at each second, but we do not directly push it into the database. Instead, we pass it through Redpanda to ensure that everything follows a certain shape. Sometimes the data is corrupted, incomplete, or missing information. For example, an amount cannot be arbitrary; it has to be a specific number. We perform all those cleanups in our code while flowing the data through Redpanda with schemas in place. The third use case is backpressure management. If you want water to do the dishes but receive a fire hose instead, we ensure that you get the right amount of quantity. We keep this very simple, and after a certain point, it becomes a day-to-day operation.
Redpanda can handle real-time financial data streams and support aggregations. For example, with stock data where each tick represents a price point per second—such as Apple stock being 252.3, then 258, then 247—you can create means, medians, and averages. All these calculations work perfectly as long as your functions are pure. You get all the data without making additional API calls or database calls, flow it through Redpanda or any Kafka instance, and perform all kinds of calculations very quickly. The database is generally the slower component. Every single time we found it was almost always the database that gets slower or the way we normalize data causing issues. Redpanda never gave us any problems. I remember RabbitMQ used to give us a lot of problems, but all those problems are gone with Redpanda.
The video example I shared involves webcams. If you consider webcams as IoT devices, we use them that way and it just works. It is not real-time in the strictest sense, but it is almost real-time, which is good enough for our case.