RabbitMQ & LavinMQ by CloudAMQP

My main use case for CloudAMQP involves using it in various ways because it is a versatile technology, but the two main ones are as a publish-subscribe technology, using stream-type queues, and as FIFO queues for direct and asynchronous communication between two or more microservices.

Between these two use cases, I consider both to be critical for my company's activities, because even if the publish-subscribe, the streaming one, can be considered a secondary use case, it has actually allowed us to cut costs and also reduce the number of microservices that were handwritten, so to speak, basically reinventing the wheel. Perhaps the second is more critical, the FIFO queues, because they made it possible to connect different data pipelines efficiently, particularly using the producer-consumer approach, even more complex ones, avoiding data loss thanks to resilience, and allowing us to scale very effectively even with thousands, almost millions of data points per second.

A real situation where CloudAMQP made a difference in my architecture is in the producer-consumer and subsequent data lake pipeline of our main data ingestion engine. In particular, we did not use RabbitMQ, we used its variant LavinMQ, which is written in Rust, so the compiled language is much faster. In this case, we used the hash-type exchange, which in turn allowed us to distribute the load at the LavinMQ application level across many more data, making it even more scalable. Besides adding resilience, it also made it even more scalable because the producer and consumer were separated. The simpler producer receives the data but must be sure that they are saved and not lost. The consumer, which is obviously heavier, has to be sure that the data it receives is correct and that it is then subsequently handled by the engine that inserts the data into the data lake.

CloudAMQP has had a positive impact on my organization in terms of efficiency, reliability, and cost reduction. Before working with CloudAMQP we had the AWS solution, specifically we were using SQS only, because streams with SNS were not being used yet. But that had a really high cost. Instead, with AMQP, even without going on-premises yet, we already had benefits in terms of the versatility of the services and in terms of costs.

The versatility and cost reduction have influenced my day-to-day activities and business outcomes significantly. I have seen faster processes and more autonomy for teams, or other tangible benefits. Having a single product to work with was an advantage, because people from various teams only had to know one product rather than, as in this case, SQS and then learn other products to solve tasks that AMQP already solves by itself. This resulted in faster processes, more streamlined operations and, indirectly, also in terms of costs. By reducing costs on one side, we could afford to evaluate other technologies for other purposes that then kept us within budget.

The best features offered by CloudAMQP include various aspects that led me to prefer it over other solutions, including managed cloud solutions like SQS, SNS, and Kafka. Above all, the fact that it can be easily managed from the cloud or hosted locally is valuable. At the beginning we started from the cloud, but the price-to-quality ratio was not that high, meaning it is a very good service, of course, but if it has to handle many data points per second, really a lot, we are talking about tens, hundreds of messages per second, then we really have to pay a lot. The same solution is also offered on-premises, so we can, as we did, install it on our Kubernetes cluster and use it easily. Its versatility is also appreciated, because besides this aspect, it also offers many different features. For example, AWS offers you Event Stream, SQS, SNS, so different services that obviously serve their business to earn more. Instead, CloudAMQP offers everything together and so with a single solution, we are able to address many different use cases.

The possibility of implementing CloudAMQP both in the cloud and locally has affected my architectural choices significantly, especially when still talking about the set of microservices that manage data ingestion and data intelligence. Previously, the solution involved using CloudAMQP and microservices that ran on individual servers, so it was more complex to manage, but we did not have to manage the message broker itself. It was actually very straightforward to manage, thanks to the API, thanks to the accounts, virtual hosts, and policies, which are very thoughtfully designed, allowing us to grant access to applications and different people in a secure way. I found this mirrored one-to-one on-premises, and I really appreciated that, because other services heavily restrict on-premises hosting, trying to push the user to work in the cloud. Working on-premises on Kubernetes, I was able to spend much less, the company was able to spend much less while remaining loyal to the product. This is obviously a positive thing, because even if CloudAMQP, or rather the company that offers the service, may not earn directly from us, indirectly it does, because we will always speak positively of this product.

CloudAMQP could be improved in some areas, though it seems pretty complete to me, also because there are plugins, so in case of a missing feature, which so far has never happened to me, you can develop them or look for them in this plugin marketplace. Something from a graphical point of view could be updated, which has always remained a bit outdated, but nothing major besides that.

In terms of necessary improvements regarding the user interface or other less obvious aspects, I think it is not very straightforward to take advantage of the charts to understand what happened further back in time, because the visualization really limits the data, showing only the most recent ones. Log management is also not particularly intuitive, which in my opinion should be reviewed.

I have been using CloudAMQP for 4 or 5 years, basically since I started working for my current company.

CloudAMQP is very stable in my experience.

Regarding scalability of CloudAMQP, I have always had a good experience. The only thing is that if you reach the load limit provided by your subscription, the server stalls and you risk losing data.

In my experience, CloudAMQP's customer support is very fast. We have dealt with them and we have always had quick and precise responses. I rate customer support a 10, since I have never had problems with them.

Previously, we were using AWS SQS, but the problem is that it is limited only to producer-consumer architectures and not streaming. Whereas AMQP lets you use both solutions at a very low cost.

I have already obtained a return on investment because there has been a time saving by using AMQP, a single solution. I can quantify this time saving as roughly 5 hours out of a 40-hour week.

Regarding my experience with pricing, setup costs, and licenses, I think setup and license costs are not a problem. Regarding pricing, I think it is convenient for managing small amounts of data, but in my opinion it is not convenient as the volume of data increases, because you very easily end up paying hundreds or even thousands of euros per month.

Before choosing CloudAMQP, I evaluated other options, looking around at other AWS services in general, but the cost, both in terms of money and the learning curve, was high, and I also looked at Apache Kafka, but that too was not convenient.

I rate CloudAMQP a 9 on a scale from 1 to 10. I give it a 9 and not a higher score because it has many strengths and in my opinion no weaknesses. I would only give a 10 after the graphical aspects are improved in a minor way, but especially when log-related aspects are made a bit more streamlined, without necessarily having to move to other log visualization and aggregation tools such as Splunk.

We use the on-premises solution on a Kubernetes cluster.

For the rest of the architecture, we use both on-premises solutions and public clouds. CloudAMQP, specifically LavinMQ, runs on our Kubernetes cluster.

The cloud provider I mainly use for these public cloud solutions is Amazon Web Services (AWS).

My advice to other people who are considering using CloudAMQP is to first assess why they need it and what volume of messages they have to handle, what the requirements of the application or context are, and then evaluate whether to move forward. I would tell them to definitely go with CloudAMQP.

We used CloudAMQP to build the entire messaging layer of a SaaS platform whose goal is to be a platform similar to Power BI, in an agnostic sense, for analytics. At the same time, it's also the platform we use to generate some packaged offerings from it. Both of these ways of contracting our platform use CloudAMQP through the MQTT protocol to enable asynchronous communication for many of our processes or flows, including data ingestion flows and large-scale data ingestion, with the purpose of receiving what needs to be done in ingestion, as well as internal communication within the application—between nodes of a cluster, or between tenants of a cluster, or even something more cross-cutting that serves as messaging and notification between several clusters and the cluster responsible for processing that message and handling what needs to be done, or something that can be a broadcast, a fan-out, so that one of the nodes of a cluster, of one of the clusters in some environment, can process what we want to process according to the flow we may implement.

The main use case for CloudAMQP is data ingestion, which became the core of analytics ingestion. Within data ingestion, there is a specific flow, which is the moment when we really need to do ingestion and reprocessing of data for all tenants of a cluster, for example, and this goes through a broadcast processing step where CloudAMQP ends up being our means both to control all these concurrent executions, based on a limit that we establish with the consumers that can keep listening to certain queues, and at the same time also becomes a major enabler for us in terms of resilience, in case our applications go down. This allows us to have a more resilient flow to deal with this and not lose messages that signal something that needs to be done or data that needs to be ingested.

CloudAMQP became the main messaging tool for our applications, or at least for the analytics application and the offerings developed on top of it. That's the main impact.

In terms of time saved after adopting CloudAMQP, not depending on the SRE area so that we could use this messaging tool integrated into our applications was already a gain, because it reduced the bureaucracy of having to request from an SRE team to make available whatever other messaging solution would have to be managed on AWS, which is our main cloud provider. As we analyzed the throughput of processing, of dealing with the volume of messages that we needed, we were also very satisfied not to see many bottlenecks related to the tool or the service itself. Another positive point is that if we contract LavinMQ, we can use this RabbitMQ variant as a container for local development or also to use in our test suites. That was fundamental so we wouldn't have to depend on the SaaS platform for something at the development level.

The best features that CloudAMQP offers are that even for developers, I found its operation to be very transparent and it requires very little involvement from dedicated SRE teams in the company. If they allocate a budget for a team, we can sign up for CloudAMQP and already start developing applications or features without a very strong dependency on the SRE area within the company. That was very interesting.

CloudAMQP offers us the ability to contract LavinMQ, which is a variant of RabbitMQ. It proved to be very lightweight and with very interesting performance when there's a high flow of messages, both being sent to queues and being consumed, without problems in this consumption.

One feature that we used in another application, which was interesting, was being able to pause queues. That helped a lot for another CloudAMQP use case, more specifically for a LavinMQ feature offered by their platform. This feature was very interesting because we could control what was being consumed in a queue, being able to pause it and then unpause it at times when we didn't want any processing to happen.

CloudAMQP's proprietary features, such as queue pause, or pausing a queue, when you contract LavinMQ, cause any application that is based on this service to become locked in. They would need to work on something that would make this LavinMQ-based service as little lock-in as possible, maybe even adding this to the RabbitMQ specification itself or something similar. Beyond that, the pricing is an area for improvement. They do not offer, as far as I remember, a pay-as-you-go plan, which would be a way for me to spend according to what I consume, without being too tied to a specific plan pricing tier. If you choose a certain plan and later want to upgrade, you have a certain monetary jump to make that upgrade. If you have a higher demand for volume or need more resources, your upgrade jump can be very large, and that can, when you only need a bit more resources, cause a cost increase that could be improved.

We have been using CloudAMQP for about three years.

During the time we had CloudAMQP in operation, it proved to be stable.

CloudAMQP offers excellent scalability; however, a drawback is that upgrading your plan can double the cost even when you only need a slight increase in capacity.

We practically didn't need to reach out to CloudAMQP support.

We used managed Kafka on AWS for the same purpose, so we migrated from Kafka to AMQP using CloudAMQP because we were using Kafka more for messaging purposes. CloudAMQP as a service showed that it would be better to make this migration viable without having a very strong dependency on the company's SRE team. Which is what ended up happening: we didn't need to involve or have SRE accompanying us in order to contract and roll out AMQP usage via CloudAMQP.

The main indicator we observed after using CloudAMQP is fewer people needing to be involved in this operation, practically by itself, without needing to add additional resources, only needing to be monitored when there is or when we need to integrate some alert to monitor a peak scenario.

The prices are attractive if you want to start rolling out CloudAMQP in your production environment, for your products that you want to make available in production. However, there's that detail of the jumps between plans if you need to upgrade. But initially, they have attractive prices.

AWS was our provider, and CloudAMQP came as an alternative to what AWS offers as a managed service in this AMQP line. We evaluated managed RabbitMQ itself, but what caught our attention with CloudAMQP is that it supports LavinMQ, which is a solution and tool that the company itself developed, and that's what attracted us because it proved to be lighter than RabbitMQ itself. That motivated us to try both the service and LavinMQ underneath it.

I rate CloudAMQP as eight based on the points that need improvement. I chose this score because of the points I brought up that still need improvement on the plans side, and also due to the lock-in that ends up being generated, which is a consequence of an enhancement, but you need to consider that as relevant and think very carefully if you're really going to adopt this solution.

I stayed more on the development and operation side, monitoring queues and capacity increase needs.

We use CloudAMQP in the same region we use on AWS as our public cloud. We also use it on-premises, using the LavinMQ container, which is part of and offered by the same company.

CloudAMQP is a good starting point as a solution based on the AMQP protocol, as long as you don't have a need for high availability or very high capacity to start using a messaging service.

My main use case for CloudAMQP is sending messages through exchanges and different queues, using different features. In this project, the idea was to send messages through exchanges and queues to create an infrastructure, a common core infrastructure, so the idea was to connect big ships and vessels to the headquarters.

Since the ships go through the ocean for weeks or months, they might lose internet connection for days or even a week. CloudAMQP provides the perfect way of sending messages, and once the ship regains access to the internet, the messages will get back to the endpoint that they were supposed to reach. The option that messages can be requeued after a certain point is a critical part of our design of our infrastructure.

Once we created this project, the company is now relying on this infrastructure that our team created, which is being used by other teams in the organization. There were different projects and different teams using various methods of communicating and sending messages, but our approach allowed us to create a common infrastructure that was reliable and faster, so that was a big success.

The best features that CloudAMQP offers include the ability for messages that fail at the beginning to go through requeue or wait periods, so after a certain time, they will be sent back to the exchange to be retried again, which was a big part of that. The requeuing and retrying feature was a critical part of the design of this common infrastructure for our company, making our infrastructure reliable and actually usable for scenarios where the ship loses internet for days or hours.

Also, the visual part is something I find important, as it allows me to see the length of the queue and the speed of processing messages in the queue, which is really important for me as a QA Engineer to debug what is going on if the messages are redirected to the queue and back to the exchange.

A better UI could improve CloudAMQP, allowing the basic length of the queue to be visualized along with a graph to show different times for a day, a week, or a month, enabling me to see how many messages were in the queue during specific times or ranges. Having options in the graph and the history of the messages would enhance my experience.

I chose nine because the main reason it is not a ten is simply that it could have more options in the graph and the history of the messages.

I would rate CloudAMQP a nine on a scale of one to ten.

I definitely advise using CloudAMQP, and I think running good performance testing is necessary to fine-tune the connectors through AMQP for your business side and the programs that are running to send and receive messages.