What Is a Message Broker?

A message broker is a software component that enables services and applications to communicate using messages. Modern applications are designed with multiple independent software components that must coordinate over a distributed architecture to complete a common task. A message broker provides a predefined message structure and mechanism so multiple components can send and receive messages without being online at the same time. Software developers use message brokers to provide reliable, persistent, and asynchronous message delivery.

A message broker acts as an intermediary interface for reliable message storage, transmission, and delivery. It allows developers to overcome messaging challenges and benefits software teams in various ways.

Persistent messaging

Message brokers ensure that all messages are safely delivered to their recipients. When application services send messages to other software components, some deliveries may fail because of network issues or recipients going offline. Message brokers store undelivered messages in persistent storage and resend them through a dead letter queue mechanism. This ensures recipients eventually receive messages addressed to them when the messaging system or recipients recover from errors.

Improved system performance

An application’s performance degrades if overloaded by excessive message processing. Consider a video processing app that receives messages like “video recording started”, “video upload started”, and “video upload finished” from millions of users. If the app stores, formats, queues, verifies, and waits for each event message, it will have minimal capacity for its primary task—video processing. A message broker facilitates communication between the app and interdependent services, improving system performance.

Asynchronous messaging

Asynchronous messaging allows applications and services to exchange messages without requiring them to always be network-connected. It allows developers more flexibility, such as putting services in sleep mode and waking them at specific intervals to send or receive messages.

Message brokers allow software developers to scale the services they build without being constrained by complex messaging routes. For example, you can use Java Message Service to decouple messaging from an application’s core functions. We share more ways of using message brokers in building modern applications below.

Manage long-running tasks

Some tasks require a longer time to complete, and applications can’t afford delays that may compromise user experience. For example, consider an audio processing software that requests an external service to transcribe a recording. Without a message broker, the app waits for the request to be processed and completed before tending to subsequent tasks. As a result, users can’t perform any other actions until the app receives a response from the transcription service. A message broker allows the app to delegate time-consuming tasks to a dedicated message queue. Instead of waiting indefinitely for results, the app can continue to handle other user requests. Once the transcription is completed, the app retrieves the result from the message broker and notifies the user accordingly.

Restructure monolithic apps with microservices

Monolithic apps are software developed from a single codebase. As developers introduced additional features, system complexity became an issue, which called for a switch to distributed systems powered by microservices.

Microservices are small software components that complete a specific function but work together to achieve larger goals. For example, a food-delivery app uses different microservices for payment, fleet tracking, and restaurant communication.

Developers use a message broker to ensure microservices exchange data efficiently in complex routing scenarios. The brokers are a central point where every microservice can publish or subscribe to get up-to-date data.

Develop transactional systems

Transactional systems process tasks in a specific sequence to avoid duplications. Every service responsible for completing the transaction retrieves messages addressed to them, processes the underlying data, and publishes the results to the broker. The broker then deletes consumed messages upon completion.

Establish communication for connected devices

Message brokers allow point-to-point messaging and asynchronous processing amongst machines, robots, the Internet of Things (IoT), and other connected devices. These devices must exchange data to operate reliably, update databases, enable analytics, and perform other functions. By connecting to a message broker, devices can easily send messages to different software components using a supported formal messaging protocol, such as RabbitMQ.

Enable mobile notifications

Mobile apps must be able to receive and display push notifications, even when not actively running on devices. With a message broker, mobile apps can pull messages from the topics they subscribe to without constantly checking with the backend services.

A message broker allows interdependent services to send and receive information through a centralized distribution system. It streamlines information flow by enforcing specific rules, sequences, and data structure.

Key architectural components

Let’s visit the basic architectural components involved to understand how message brokers work.

Producer

The producer is the app that sends messages to one or more recipients. For example, a camera app that stores captured images in the cloud is a producer.

Consumer

The consumer is an app that retrieves or receives messages from the message broker. Depending on the broker type, one or multiple consumers may retrieve the same message a producer sends.

Message queue

The message queue is a pipeline that stores messages until consumers read them. Depending on the requirements, the message broker might sort or format stored messages before sending them to consumers. In the event of consumer failure, messages are retained in the queue.

Exchanger

The exchanger creates topics to group similar types of messages together. This way, consumers can subscribe to specific topics instead of retrieving and filtering all messages in the queue.

Point-to-point messaging

In a point-to-point architecture, one message producer communicates directly with one consumer. The producer sends a message to the message queue, which the broker stores until a consumer reads it. Then, the broker deletes the message from the queue. Even if several consumers connect to the message queue, only one consumer receives the message. Point-to-point messaging is suitable for financial transaction processing to avoid duplication. For example, you can assign different services for verification, fund transfer, and notifications.

Publish/subscribe messaging

The publish/subscribe (pub-sub) messaging architecture comprises loosely coupled producers and consumers. In this setup, producers are known as publishers and consumers as subscribers. Together, they form a one-to-many relationship. Each publisher stores messages on topics that multiple subscribers can retrieve. Topics are different subjects that the broker uses to categorize different messages. For example, you can publish messages on politics, economy, and sports topics when developing pub-sub messaging for a news app. The broker would route messages belonging to a specific topic to subscribing recipients.

Applications use different messaging components when exchanging data. Here’s how message brokers compare to others.

Message brokers vs. message queues

Message queues are data structures that hold messages that producers store in a specific sequence. Messages are queried in a first-in, first-out (FIFO) order and remain in storage until their intended recipients read them. Sometimes, message brokers use message queues as temporary, fault-tolerant storage to provide a reliable messaging system to software applications.

For example, Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing platform for microservices, distributed systems, and serverless applications. It lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Message brokers vs. event streaming platforms

Event streaming platforms are managed services that allow software applications to process a continuous stream of application events. Events are sorted into topics consumers can subscribe to. While sharing data in a distributed event streaming platform, distribution principles are similar to message brokers; however, streaming platforms focus on high-speed storage and recoverability. They replicate messages with partitions and store them on different physical servers. In contrast, message brokers are more concerned about delivering messages to their intended recipients. Therefore, an event streaming platform might not have the tracking and message-resending features a message broker does.

Message brokers usually delete stored messages after they’re read by all recipients, but streaming platforms may retain them indefinitely. When an event streaming platform receives a new message, it appends to the growing queue of messages.

Consumers can filter the type of messages they receive when subscribing to topics on a message broker. However, all subscribers receive the same messages and must programmatically filter them within the application.

Message brokers vs. message bus

A message bus is a messaging middleware that bridges different hardware, services, and software on a computer system. Also known as an enterprise service bus (ESB), the message bus restructures messages sent by other components with a standard protocol. ESB is commonly used in service-oriented architecture (SOA). SOA is a software development approach that uses different services to build applications. Compared to message buses, message brokers are a more agile version of message distribution systems. Developers prefer message brokers because they are scalable, flexible, and less expensive to maintain.

Amazon MQ is a fully managed broker service that allows you to set up, streamline, and manage popular message brokers effortlessly. With simple steps, you can connect producers and consumers built in different languages to Amazon MQ and exchange high-throughput information. For example, you can:

• Migrate on-premise applications to AWS with a network of brokers without tedious configuration.

• Ensure fault-tolerant, highly recoverable endpoints by exchanging messages reliably across AWS availability zones.

• Integrate with other services like Amazon CloudWatch and AWS CloudTrail for monitoring all brokers, queues, and topics your applications are connected to.

Get started with message brokers on AWS by creating a free account today.