AutoMQ Cloud - The Low-Latency, Diskless Kafka® on S3 (Pay as You Go)

Our primary use case is serving as a real-time data backbone within our cloud-native architecture. Specifically, we rely on AutoMQ Cloud to handle high-throughput user behavior logs and order transaction data. Previously, when using traditional Apache Kafka, storage scaling and configuring cross-AZ high availability were constant pain points for our DevOps team.

After migrating to AutoMQ Cloud, the key benefit for us is its compute-storage separation architecture. Now, during traffic spikes (like major sales events), we can independently scale up computing resources to handle write pressure, while the cost of storing massive amounts of historical data has dropped significantly thanks to S3 integration. Simply put, it has freed us from the heavy operational burden of managing Kafka, allowing us to focus much more on developing core business logic.

The biggest improvement AutoMQ Cloud has brought to our organization is a drastic reduction in operational complexity and a significant optimization of our cost structure. From an operations perspective, we used to spend a lot of engineering hours manually intervening and monitoring data recovery whenever we faced Kafka cluster rebalancing or broker failures. With AutoMQ's S3-based shared storage architecture, scaling nodes up or down has become incredibly lightweight and fast.

Now, when traffic spikes occur, we can complete scaling within minutes. Our Recovery Time Objective (RTO) has dropped from hours to minutes, giving us a quantum leap in overall system stability. On the cost side, thanks to hot/cold data tiering and the low-cost nature of object storage, our overall messaging infrastructure costs have actually decreased by about 40%, even though we've extended our data retention periods. This has been a huge win for our business, especially since we need to retain logs long-term for analytics.

For us, the most valuable feature is undoubtedly the compute-storage separation architecture based on object storage (S3). As developers, our biggest headache with traditional Kafka was always the 'data shuffling' process during scaling—it was time-consuming and often triggered network storms. AutoMQ leverages cloud-native advantages to make compute nodes stateless. This means scaling up no longer requires waiting for data replica synchronization; it's almost instant.

This rapid elasticity fits our fluctuating business traffic perfectly. Additionally, full compatibility with the Apache Kafka API has been a huge plus. During our migration, we didn't have to change a single line of our application code. We simply swapped out the client connection addresses to complete the switch, which drastically reduced our trial-and-error costs and migration risks.

AutoMQ Cloud is already excellent in its core messaging services. If I had to point out areas for improvement, I would suggest enhancing the managed integration for Kafka Connect and strengthening its streaming data lakehouse capabilities.

I have been using the solution for 3 years.

We have not encountered any stability issues.

There have been no scalability issues.

The customer service is satisfactory.

Before migrating to AutoMQ Cloud, we were using Amazon MSK (Managed Streaming for Apache Kafka). While MSK saved us from maintaining the underlying infrastructure as a fully managed service, it still retained the architectural limitations of traditional Kafka with local disk-based storage. In practice, whenever we needed to scale up during traffic peaks, MSK would go through a lengthy partition rebalancing and data synchronization process. This was not only time-consuming but also prone to impacting the stability of our live services. Additionally, as our data volume grew, the storage costs associated with MSK's EBS volumes became prohibitively expensive.

We ultimately decided to switch to AutoMQ because we were drawn to its cloud-native compute-storage separation architecture. By leveraging S3 for durable storage, AutoMQ completely eliminates the slow scaling pain points of Kafka, allowing us to achieve true rapid elasticity in the face of fluctuating traffic. At the same time, offloading cold data to S3 slashed our storage costs by several times. For a tech team like ours that prioritizes extreme elasticity and cost-efficiency, AutoMQ is simply a more modern and cloud-aligned choice than MSK.

The initial setup was straightforward.

Our implementation was handled by an in-house team.

The solution has delivered a positive return on investment.

Regarding pricing, my advice is: don't just focus on the base hourly resource rates; instead, evaluate it from the perspective of Total Cost of Ownership (TCO).

One final piece of advice for teams considering a migration from traditional Kafka: go ahead and give it a try with confidence—the migration cost is genuinely very low.