Getting started with MQTT retained messages for AWS IoT Core

AWS recently announced the general availability of MQTT retained messages for AWS IoT Core. This feature allows you to store a single message per a given MQTT topic for delivery to any current and future topic subscribers. Creating a retained message simply requires setting a retained flag when publishing to let the AWS IoT Core broker know to save it. At that point the message is delivered to all current subscribers as normal, and also any new devices that subscribe to that topic in the future. Also included in this launch are new and updated APIs to help manage retained messages. Retained messages can be used in all regions where AWS IoT Core is available.

In this post, we provide an overview of this feature, share some use cases to keep in mind when designing your next IoT project, and provide a guide on how to get started both in the AWS IoT console and programmatically.

Overview

Retained messages are defined in the MQTT 3.1.1 specification, and are created by setting the retained flag to ‘True’ when publishing a message. This flag is supported by the AWS IoT Device SDKs as well as most MQTT 3.1.1 compatible clients. A retained message is delivered to any currently subscribed clients just like a normal message, and it will also be automatically sent to new clients upon subscribing to that topic. As only one retained message can be saved per topic, any future retained messages that are sent will replace an existing retained message if there is one. AWS IoT Core will store retained messages for up to three years after the last time they were updated or accessed. Publishing a new retained message with an empty payload can be used to remove the currently retained message from a particular topic. Managing these messages can be done through the new Retained Messages page within the AWS IoT console. This page can be found under the “Manage” tab where you can view all of the currently retained messages in the current region as well as update them. Management can also be performed over HTTP through the use of the new ListRetainedMessages, GetRetainedMessage, and updated Publish APIs.

Use cases for retained messages

1. Distributing configuration settings for devices

One reason to use retained messages is to store configuration settings for one or more devices. This may be helpful when finishing up a bootstrapping process after a newly provisioned device first comes online, or as a discovery message for when devices connect to an IoT application and need to be informed of its capabilities. Devices or other clients could also update that retained message with the latest configuration settings as needed. When using this approach on a per-device basis where each device has its own unique configuration topic, then it is worth considering Device Shadow for AWS IoT Core. One difference would be that a retained message will be automatically delivered when subscribing versus subscribing to Device Shadow updates and deliberately requesting the current state if needed. To understand all of the key differences between the two, check out Using MQTT retained messages which includes a comparison table to help you choose the right tool for your particular use case.

Another reason to use a retained message is to broadcast a shared configuration to multiple devices. For example, if you have a configuration to send to a group of devices, then having the applicable devices subscribe to a shared topic would ensure that all current and future subscribers to that particular topic will have the latest version of their shared configuration. Unique device configurations could still be managed in either a Device Shadow or retained on a unique topic, while any shared or group settings could be retained within the shared topic.

For example, a user publishes a message to a topic that multiple devices subscribe to. Two online devices receive the message. With the retain flag set to ‘True’, the AWS IoT Core broker saves the message for future devices that subscribe to that topic. If a third device is currently offline, the message sent by the user is retained, so the offline device will automatically receive it once it connects and subscribe to that topic.

2. Keeping track of the last known state of devices

Another way to use retained messages is for storing the current state of one or more devices. For example, a device could periodically publish its current state to a unique topic and set the retain flag to ‘True’ each time so that the latest state is always stored by the AWS IoT Core broker. If another client, such as a cloud service or customer-facing application needs the device state, it can just subscribe to the particular topic and automatically receive it instead of waiting for the device to publish an update. As with the first device configuration example, this particular pattern of using a unique retained message topic per device to maintain state shares similarities with Device Shadows, so please review the comparison table in Using MQTT retained messages to help choose the best option for your particular use case.

For maintaining the current connectivity state of a device, use the ‘Will Retain’ flag so that if a device has an ungraceful disconnect, the retained message of a particular topic is updated so that current and future subscribers can know the current connectivity state of that device. This way, if a device has an ungraceful disconnect, any other devices that subscribe to that topic before it reconnects will automatically be notified that it is currently offline.

Using retained messages in the AWS IoT console

To test out retained messages from within the AWS IoT console, we’ll use a combination of the MQTT Test Client as well as the new Retained messages page that can be found under the Manage section. It’s important to note that retained messages have associated AWS IoT Core policy actions that allow you to control access to this feature. Check out Example AWS IoT policies to see examples of how to authorize access to retained messages. First, we’ll use the MQTT Test Client to publish a retained message to a test topic. Afterwards, we’ll see how to view and update that message from the Retained Messages tab. Finally, we’ll go back to the MQTT Test Client, subscribe to that test topic, and see the delivery of the retained message right from our browser. Let’s get started.

Step 1: Publish a test message with the MQTT test client

1. Log in to the AWS Management Console, select Services along the top and navigate to AWS IoT Core.
2. Select Test in the left-hand navigation and then MQTT test client.
3. Select the Publish to a topic tab and enter the topic name that you’d like to use for your test. For this example, imagine we’re making smart home products and want to set up a topic per customer that stores account settings, which influence device settings such as opt-in usage of certain settings or premium subscription services.
   1. Enter “users/settings/user01” as the topic.
   2. Enter {“account_type” : “standard”} for the message to include the customer’s current account type.
4. Expand the Additional configuration section and choose the Retain message on this topic checkbox.
5. Select Publish.

Step 2: View the retained message within the Retained Messages page

1. Now that we’ve published a retained message, select Manage in the left-hand navigation and then Retained messages. From here, you’ll see a list of topics that currently have retained messages.

2. Choose the topic that you published a message to in Step 1. From here, you can see the retained message that is currently being saved on the topic.

Step 3: Subscribe in MQTT test client

Now that we’ve confirmed our test message has been retained successfully, let’s see it in action.

1. In the left-hand navigation, choose MQTT test client under the Test section.
2. Choose the Subscribe to a topic tab.
3. In the Topic filter field, enter “users/settings/user01”.
4. Select Subscribe. You should see the retained message pop up on your screen.

Using retained messages within your projects

Publishing retained messages from within your device and application logic is supported by the AWS IoT Device SDKs as well as most MQTT clients since retained messages is a standard feature of the MQTT 3.1.1 specification. One way that retained messages for AWS IoT Core differ from the specification is that in order to receive a retained message stored on a particular topic, you’ll need to match the topic filter exactly without the use of any wildcards (i.e. #,*).

Publishing a retained message using the AWS IoT Device SDK

Let’s look at an example of how we can publish a retained message by using the AWS IoT Device SDK v2 for Python. Here is a snippet from the MQTT Pub/Sub sample that is responsible for publishing a message:

mqtt_connection.publish(
topic=args.topic, 
payload=message_json,
qos=mqtt.QoS.AT_LEAST_ONCE)

While this example defaults to QoS 1, QoS 0 is also supported for retained messages. Regarding setting the retain flag, if we look at the API documentation, we see that setting it when using the publish class just requires adding the parameter retain=bool.

So, to publish a retained message with the AWS IoT Device SDK v2 for Python, add that parameter to the end of the snippet like this:

mqtt_connection.publish(
topic=args.topic,
payload=message_json,
qos=mqtt.QoS.AT_LEAST_ONCE,
retain=True)

As a subscriber, retained messages can be differentiated from standard messages by checking the same flag.

To test this, publish a new retained message to the topic “users/settings/user01” that says “hello from python”. The example can be run using the following command line:

python3 pubsub.py --endpoint YOUR-ENDPOINT \
--port 8883 \
--cert YOUR-CERT \
--key YOUR-KEY \
--root-ca ROOT-CA \
--client-id YOUR-CLIENT-ID \
--topic users/settings/user01 \
--count 1 \
--message 'hello from python!'

In the AWS IoT console, verify that you now see this new message reflected in the Retained messages section.

While we just looked at Python in this example, check out other AWS IoT Device SDKs which also support the retain flag out of the box.

Conclusion

With MQTT retained messages for AWS IoT Core, you can save messages on topics for automatic delivery to current and future subscribers. Creating and managing retained messages can be done from within the AWS IoT console, or through the AWS IoT Device SDKs as well as the new ListRetainedMessages, GetRetainedMessage, and updated Publish APIs. Check out AWS IoT Core endpoints and quotas for related throttling and service limit information. This feature is available now, in all regions where AWS IoT Core is available. To learn more, check out the Using MQTT retained messages documentation.