This post is contributed by Mithun Mallick, Solutions Architect and Christian Mueller, Solutions Architect

Message-oriented middleware (MOM), or message brokers, are the backbone that integrates business critical applications in many industries. MOMs are used to integrate systems like inventory management, payment systems, and CRM systems. They are also used to orchestrate order-processing workflows across multiple systems, or integrate modern web applications with legacy backend applications.

Some of the most commonly used MOMs in the market are IBM MQ, TIBCO EMS, Rabbit MQ, and Apache ActiveMQ. These systems are often costly to maintain and can have high licensing costs.

Our customers often tell us they want to migrate their applications to the cloud but are hindered by the heavy-lift involved in migrating their MOM systems. They are concerned that migrating their MOM is difficult, and that they must migrate all interconnected systems in one step.

In this post, we show how to build a bridge solution from on-premises IBM MQ to Amazon MQ, to migrate your applications in a step-by-step manner. Amazon MQ is a managed message broker service from AWS that makes it easy to set up and operate message brokers in the cloud.

We explain the steps to move the producers (senders) and consumers (receivers) in phases from your on-premises to the cloud. This process uses Amazon MQ as the message broker, and decommissions IBM MQ once all producers/consumers have been successfully migrated.

You can migrate applications without disrupting existing business systems, and take advantage of the agility, flexibility, and reliability of the cloud. The same pattern can be used if you are using another MOM system, like TIBCO EMS or Rabbit MQ (which supports JMS/NMS, AMQP, or MQTT).

In this post, we show how the phased migration approach helps you mitigate the risk involved a ‘big bang’ cutover. The goal of the solution is to enable an incremental migration of your on-premises applications and MOM to the AWS Cloud. It also replaces your message broker with Amazon MQ. The solution is shown in the diagram below:

You can follow the step-by-step instructions in the GitHub’s README file for your own migration.

Initial state and pre-requisites

In the initial state, you are running producers and consumers connected to an on-premises IBM MQ broker.

The first step is to facilitate a bridge from IBM MQ to Amazon MQ. Given that IBM MQ is running on-premises, the bridge solution must either exchange messages over the internet by setting up an AWS VPN tunnel, or by using AWS Direct Connect to establish connectivity with the AWS Cloud.

To ensure confidentiality and security for your message exchange, we recommend using AWS VPN. If you have low latency requirements to forward your messages, using AWS Direct Connect is the recommended way.

To complete the hands-on portion of this blog, you must have access to an IBM MQ broker. If you don’t have access, you can provision an IBM MQ broker, running on a Docker container in AWS Fargate.

Once the connectivity is established, you must set up an Amazon MQ broker from the AWS Management Console, or using an AWS CloudFormation template. There is a template provided as part of this post.

Step – 1

Begin by deploying some consumer applications on AWS. These consumers can be new applications or additional instances of applications already running on-premises. You configure these consumer applications to receive messages from Amazon MQ. At this stage, message producers are still on-premises sending messages to the IBM MQ broker.

Next, bridge from IBM MQ to Amazon MQ using a proxy pattern. The proxy pattern is technology-agnostic, and you implement the pattern using Apache Camel to build a JMS bridge. Apache Camel is an open source integration framework for implementing Enterprise Integration Patterns. Apache Camel includes JMS components that easily connect with IBM MQ and Amazon MQ.

In this step, you build an Apache Camel route to consume messages from IBM MQ, and forward to Amazon MQ. Here is an example from the camel-context.xml file, which defines the configuration:

<route id="ibmMQ-to-amazonMQ">
<description>Camel Route from IBM MQ to Amazon MQ</description>
<from uri="ibmMQ:queue:DEV.QUEUE.2?concurrentConsumers=5"/>
<inOnly uri="amazonMQ:queue:DEV.QUEUE.2?preserveMessageQos=true"/>
</route>

This Apache Camel route defines how messages from the producer applications connected to IBM MQ move to Amazon MQ. In this example, there is one sample route but you may have many routes in your production use-case.

Apache Camel is deployed as a Docker container running on AWS Fargate as compute engine for Amazon Elastic Container Service (ECS). ECS is a container orchestration framework that manages the deployment of the containers, and runs them in a highly scalable manner.

AWS Fargate eliminates the heavy lifting of scaling the underlying virtual machines for your ECS cluster. By defining the desired capacity of AWS Fargate tasks, it introduces self-healing capabilities to the JMS bridge. AWS Fargate tracks the number of healthy tasks, and creates new tasks automatically if an old one is no longer available.

Now the JMS bridge and the on-premises consumers are listening on the same queue and waiting for messages. Messages sent to IBM MQ are consumed by on-premises consumers as well as the JMS bridge.  The JMS bridge forwards messages to Amazon MQ to be consumed by the applications already migrated to the cloud. You can now validate that messages are processed by applications on AWS and on-premises.

Now phase 1 of the migration is validated. You can continue to move more consumers as you get more comfortable with the availability and scalability of the bridge solution. The goal of this phase is to reach a state where messages are still produced on-premises, with some consumers running on AWS.

Step – 2

Several consumer applications are now migrated to AWS. The goal now is to move the producers to AWS. Start the migration by running a few producer applications on AWS and connect them to Amazon MQ. The Apache Camel bridge is also updated to facilitate bidirectional flow of messages.

The following configuration code shows the route, which moves messages from Amazon MQ to IBM MQ:

<route id="amazonMQ-to-ibmMQ">
<description>Camel Route from Amazon MQ to IBM MQ</description>
<from uri="amazonMQ:queue:DEV.QUEUE.2?concurrentConsumers=5"/>
<inOnly uri="ibmMQ:queue:DEV.QUEUE.2?preserveMessageQos=true"/>
</route>

At this point, messages originating from the producers on AWS have consumers on both on-premises and on AWS. You can validate the processing of messages on AWS as well as on-premises. The state can be seen in the picture as below:

Move more producer applications to AWS as you validate test results, and are comfortable with the bridge solution. Step 2 of the migration is validated once you have confirmed the results of the testing on AWS.

Step – 3

To make this JMS bridge resilient, it must scale in and out automatically, based on your current load. You can configure this by using Amazon CloudWatch metrics and CloudWatch alarms. These alarms can trigger scaling activities to scale in or out, with a fixed number of instances or a percentage-based scaling.

You can also scale out your AWS system based on the utilization of your on-premises broker by defining custom CloudWatch metrics. For example, by running a CRON script on the on-premises broker machine to periodically report the metrics such as queue depth.

At this point, this has shown the advantages that the cloud offers for running services efficiently with high availability and resiliency. The automatic scaling capabilities of Amazon ECS generate additional instances of Apache Camel containers as load increases and the queues are filling up. It also scales it in as the load decreases.

You have now established a stable and scalable bridge solution. The next step is to move all the remaining producer/consumer applications to the AWS Cloud. If you have applications that cannot move to the cloud, such as mainframe applications, these can remained connected through your on-premises IBM broker. All other applications can be migrated.

Step – 4

All producers and consumer applications have now been moved to AWS. All the messages that are sent to Amazon MQ broker are processed directly by the consumers running on AWS. The Apache Camel route to move messages from Amazon MQ to IBM MQ and vice versa is disabled.

Step – 5

The final goal is to move all application from on-premises to the AWS Cloud. Once all applications are migrated, you can decommission the Apache Camel bridge solution. All the resources deployed in the Apache Camel bridge solution are deleted, along with the automatic scaling and Amazon CloudWatch alarm configuration.

All producers and consumers are now migrated to running on AWS with Amazon MQ as their message broker.

Conclusion

In this blog, we showed how to migrate on-premises applications that are dependent on commercial message brokers to the AWS Cloud. The approach relies on a bridge solution, which is based on the proxy pattern, and is technology-independent.

The bridge provides a low risk migration of the applications in phases so that you can validate the migration and avoid any disruption to your business.

For more information on migrating to Amazon MQ and using Apache Camel, please see Migrating from RabbitMQ to Amazon MQ and Integrating Amazon MQ with other AWS services via Apache Camel.