Implementing Local Client Devices with AWS IoT Greengrass

AWS IoT Greengrass is an open source edge runtime and cloud service that helps you build, deploy, and manage device software at the edge. AWS IoT Greengrass Version 2.2.0 now includes support for local clients and devices (client devices) to locally connect to an AWS IoT Greengrass Core using MQTT. Client devices can securely send and receive messages using the Message Queuing Telemetry Transport protocol (MQTT), either to AWS IoT Core in the cloud, or communicate with other components via the Interprocess communication feature (IPC).

These new capabilities simplify the security, connectivity, and delivery of messages using AWS IoT Greengrass. This provides a seamless transition between things using AWS IoT Core for communication and the local AWS IoT Greengrass Core by using the same X.509 client certificate, MQTT connection, and MQTT topic namespace for interacting with messages.

In this blog post I describe use cases for client devices using a local AWS IoT Greengrass Core for connectivity, messaging, and interaction with IPC-enabled components. I step through the process for configuring and deploying the AWS IoT Greengrass components for MQTT, and how client devices are authorized to connect and communicate on set topics. I then demonstrate local MQTT messaging, messaging to AWS IoT Core, and IPC interoperability from a client device perspective.

The audiences for this blog post are architects or developers creating edge compute solutions who are familiar with setting up and operating AWS IoT Greengrass Version 2. If you are new to AWS IoT Greengrass or currently using AWS IoT Greengrass Version 1.x, please read What is AWS IoT Greengrass (Version 2) and step through the Getting started tutorial to familiarize yourself with the terminology and operation of AWS IoT Greengrass Version 2.

Overview

To illustrate the client device and local MQTT broker capabilities, I will use the following architecture to demonstrate client device interaction:

Reference architecture of an AWS IoT Greengrass device communicating with local client devices and AWS

In this architecture, there are two client devices, thing1 and thing2, both configured and authorized to connect to the local MQTT broker. I will refer to them as things when describing resources in AWS IoT Core, and when connecting and interacting with AWS IoT Greengrass, will refer to them as client devices. Once connected to AWS IoT Greengrass, the client devices can publish and subscribe to MQTT topics for different local use-cases such as communication with other devices, local shadows, and to AWS IoT Core topics and shadows in the cloud.

To demonstrate the different usage patterns, I show the component configuration for client devices thing1 and thing2 communicating through the local MQTT broker, how the client devices can publish and subscribe to topics mirrored in the cloud, and how they can also interact with other components via the IPC.

For demonstration purposes, overly permissive policies are used. For production deployments, please refer to the Service role permissions for core devices and the Minimal AWS IoT policy for AWS IoT Greengrass V2 core devices documentation pages for limiting permissions needed by AWS IoT Greengrass.

Deploy the client devices components

There are three client device components that make up the local MQTT environment for client devices to connect and communicate. By using these components, the client devices thing1 and thing2 will communicate locally with AWS IoT Greengrass and not require an MQTT connection to AWS IoT Core in the cloud. I will add all three of the components to a new or revised deployment and describe the use and configuration details for each.

The first component is the Moquette MQTT broker component ( aws.greengrass.clientdevices.mqtt.Moquette), which is the local MQTT broker that will be used by the client devices for native MQTT communication, and used by the MQTT bridge component for translation to and from the IPC publish and subscribe capability. This component is based on a fork of the open source Moquette Project MQTT compliant broker. The default configuration values will have Moquette bind to all network interfaces and only use the default MQTT secure SSL communication listening on port 8883. These value can be changed by the merge update process.

The next component is the MQTT bridge component ( aws.greengrass.clientdevices.mqtt.Bridge), which relays MQTT messages between client devices, local AWS IoT Greengrass publish/subscribe, and AWS IoT Core. You use this component to route messages between the Moquette MQTT broker component (LocalMqtt) and either AWS IoT Core (IotCore) or the IPC (Pubsub). I will use the following merge update to allow the local client devices to access an AWS IoT Core topic, and to allow messages published on AWS IoT Core to be received locally by the same client devices :

{
  "mqttTopicMapping": {
    "ClientDevicesToCloud": {
      "topic": "from_local/hello/world",
      "source": "LocalMqtt",
      "target": "IotCore"
    },
    "CloudToClientDevices": {
      "topic": "from_cloud/hello/world",
      "source": "IotCore",
      "target": "LocalMqtt"
    }
  }
}

In this merge update JSON document for the MQTT bridge component, I set up one mapping that listens for messages on the from_local/hello/world topic from the local MQTT broker and then publishes that message to to the same topic on AWS IoT Core in the cloud. The second mapping is the reverse where the MQTT bridge is subscribed to the from_cloud/hello/world topic on AWS IoT Core. When a message is received on that topic, the MQTT bridge component will publish it to the same topic on the local MQTT broker.

When configuring for your own use, make sure to not configure any source and target mappings that use the same topic, otherwise this will cause an infinite message loop. For example, if the topic hello/world was used in the configuration above for both mappings, a message published on hello/world to the local MQTT broker would be published on AWS IoT Core. This would then be seen as a message being received on AWS IoT Core, and then re-published back to the Moquette MQTT broker in an infinite loop. If there is request-response pattern using MQTT, use one topic for the publishing requests such as cmd/my_topic/request, and a separate response topic such as cmd/my_topic/response to subscribe for the responses. This pattern is described in the IoT Atlas.

With the MQTT bridge component merge configuration applied, the final component to be deployed is the Client device auth component (aws.greengrass.clientdevices.Auth), which authenticates client devices and authorizes client device actions. The component performs this by matching AWS IoT Core thing names to client devices, and then applying a policy that informs the MQTT broker as to what MQTT operations and topics client devices can act upon. The merge update below will reference thing1 and thing2, and then authorize them to communicate locally on a topic, and also to access the from_local/hello/world and from_cloud/hello/world topics:

{
  "deviceGroups": {
    "formatVersion": "2021-03-05",
    "definitions": {
      "MyDeviceGroup": {
        "selectionRule": "thingName: thing1 OR thingName: thing2",
        "policyName": "MyBlogPostPolicy"
      }
    },
    "policies": {
      "MyBlogPostPolicy": {
        "AllowConnect": {
          "statementDescription": "Allow connections with matching clientId",
          "operations": ["mqtt:connect"],
          "resources": ["mqtt:clientId:thing1", "mqtt:clientId:thing2"]
        },
        "AllowLocalPublish": {
          "statementDescription": "Allow client devices to publish on LOCAL test/topic",
          "operations": ["mqtt:publish"],
          "resources": ["mqtt:topic:test/topic"]
        },
        "AllowLocalSubscribe": {
          "statementDescription": "Allow client devices to subscribe to LOCAL test/topic/response",
          "operations": ["mqtt:subscribe"],
          "resources": ["mqtt:topicfilter:test/topic/response"]
        },
        "AllowSubscribeToCloud": {
          "statementDescription": "Allow client devices to subscribe to the AWS Cloud originated topic",
          "operations": ["mqtt:subscribe"],
          "resources": ["mqtt:topicfilter:from_cloud/hello/world"]
        },
        "AllowPublishToCloud": {
          "statementDescription": "Allow client devices to publish on a local topic that will be sent to the AWS Cloud",
          "operations": ["mqtt:publish"],
          "resources": ["mqtt:topic:from_local/hello/world"]
        }
      }
    }
  }
}

Verify connectivity, local, and cloud messaging

When the Client device auth component is deployed, it will perform a variety of operations to configure the Moquette MQTT broker component. First, it will create a list of thing names in AWS IoT Core that match the selectionRule, which is this case would be to include thing1 and thing2. Next, with the list of things to include as client devices, the Client device auth component will then transfer the attached thing certificates to the AWS IoT Greengrass Core and configure the Moquette MQTT broker component to use them for mutual TLS authentication. Finally, the Client device auth policy or policies are applied to the Moquette MQTT broker component. This enforces what client devices are able to connect to the local AWS IoT Greengrass Core Moquette MQTT broker (via the mqtt:connect operation) and what MQTT topics each thing can subscribe to or publish messages.

With all three components added and configured to a deployment, the AWS IoT Greengrass Core will download the newly added components and configure. You can verify the status of the deployment by looking in the $GG_ROOT/logs/greengrass.log file and monitor for client device connections, subscriptions, and publish operations, as seen in this output from the log file:

[INFO] (nioEventLoopGroup-3-8) com.aws.greengrass.device.DeviceAuthClient: Creating new session. {}
[INFO] (nioEventLoopGroup-3-8) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
[INFO] (nioEventLoopGroup-3-8) com.aws.greengrass.mqttbroker.ClientDeviceAuthorizer: Retrieved client session. {clientId=thing1, sessionId=81843a7e-b7d7-4e23-b94a-7492d2039dd6}
[INFO] (nioEventLoopGroup-3-8) com.aws.greengrass.mqttbroker.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=thing1, sessionId=81843a7e-b7d7-4e23-b94a-7492d2039dd6}
[INFO] (nioEventLoopGroup-3-8) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <thing1> to topics [MqttTopicSubscription[topicFilter=test/topic/response, option=SubscriptionOption[qos=AT_LEAST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
[INFO] (nioEventLoopGroup-3-8) io.moquette.broker.metrics.MQTTMessageLogger: C<-B SUBACK <thing1> packetID <36810>, grantedQoses [1]. {}
[INFO] (nioEventLoopGroup-3-8) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <thing1> to topics [MqttTopicSubscription[topicFilter=from_cloud/hello/world, option=SubscriptionOption[qos=AT_LEAST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
[INFO] (nioEventLoopGroup-3-8) io.moquette.broker.metrics.MQTTMessageLogger: C<-B SUBACK <thing1> packetID <36811>, grantedQoses [1]. {}

As seen from the log file, thing1 connects to the Moquette MQTT broker and subscribes to the local response topic test/topic/response and also to the topic from_cloud/hello/world that receive messages from AWS IoT Core. Besides client device entries, all MQTT operations will be logged such as publishes, connections, and disconnections.

To perform the testing, I created the Node-RED flow below with an MQTT configuration for thing1 that simultaneously publishes a message on both the local test/topic and from_local/hello/world topics. Each box represents an operation taking place, with the flow from left to right. The nodes in white are comments to describe the two different sections.

It is important understand the MQTT configuration when connecting and validating a connection to the Moquette MQTT broker from a client device. The X.509 server certificate used by the Moquette MQTT broker has special attributes that client devices use to validate the endpoint to connect, and also the signing certificate authority. While this capability is included in the AWS IoT Device SDKs, it does require additional understanding when using other MQTT clients such as Node-RED. Please refer to the Interact with local IoT devices section of the AWS IoT Greengrass documentation for more details.

Within the same Node-RED flow I also have the Subscribe to local responses… section, where thing1 is subscribed to the test/topic/response (local) and from_cloud/hello/world (AWS cloud) topics. The top flow demonstrates how messages are published, and the bottom flow receives and processes messages from the subscribed topics.

Node-RED flows for locally publishing to either local or cloud topics

With thing1 connected, clicking on the publish hello world message node sends a sample message to the two topics on the Moquette MQTT broker. The message to test/topic will be processed by the Moquette MQTT broker component and delivered to a local component (not shown, it is the operation shown by the arrow from the processed by component node). This local component will then publish a response to the test/topic/response topic to which thing1 is also subscribed. The Received message text in the debug window verifies the message was received and processed by the local component back to thing1.

In parallel, the second message to the from_local/hello/world topic is forwarded to AWS IoT Core by the MQTT bridge component. To confirm messages being sent and received to AWS IoT Core, I navigate in the AWS console to AWS IoT Core → Test → MQTT test client and subscribe to the from_local/hello/world topic, and click on the publish hello world message node again. In the AWS console, from the Subscriptions for the topic, I then see the message that was published to the Moquette MQTT broker and processed by the MQTT bridge component has been delivered to AWS IoT Core:

AWS IoT Console test client showing messages received from thing 1 via the MQTT Bridge

To test the reverse message flow from AWS IoT Core to the local AWS IoT Greengrass MQTT broker, I enter from_cloud/hello/world in the Topic name field, I then create a response in the Message payload text box, and finally select the Publish button to send the message. I see it arrive in the Debug messages pane within Node-RED configuration for thing1 subscribed to the same topic on the Moquette MQTT broker on the AWS IoT Greengrass Core.

Message sent from AWS IoT Core and received in Node-RED connected to AWS IoT Greengrass

With the verified output from both local-to-local, mapped local-to-cloud, and cloud-to-local, you now have the ability configure, authenticate, and authorize client devices with the provided Moquette MQTT broker.

Bridging IPC and local MQTT topics

The MQTT bridge component can also create mappings between the IPC and the Moquette MQTT broker for bidirectional communication. If you require IPC component communication directly to only an AWS IoT Core topic, instead of using the MQTT bridge component, use the AWS IoT Core MQTT messaging IPC service instead.

For instance, in the above example where I discuss the processed by component node, this is a local component that subscribes to test/topic via the IPC instead of MQTT. When it receives a message it then copies the content and publishes it on the test/topic/response topic along with a count of the times has been invoked since startup. To accomplish this client device interaction with the local component, I update the MQTT bridge component configuration to map the topics to and from the Pubsub mapping which is used by custom components. Here is the updated MQTT bridge component merge update with the addition of LocalMqtt to Pubsub for topic/test,and Pubsub to LocalMqtt for topic/test/response:

{
  "mqttTopicMapping": {
    "ClientDevicesToCloud": {
      "topic": "from_local/hello/world",
      "source": "LocalMqtt",
      "target": "IotCore"
    },
    "CloudToClientDevices": {
      "topic": "from_cloud/hello/world",
      "source": "IotCore",
      "target": "LocalMqtt"
    },
    "DevicesToIpc": {
      "topic": "test/topic",
      "source": "LocalMqtt",
      "target": "Pubsub"
    },
    "IpcToDevices": {
      "topic": "test/topic/response",
      "source": "Pubsub",
      "target": "LocalMqtt"
    }
  }
}

With this configuration deployed to the AWS IoT Greengrass Core, a client device can use MQTT to send and receive messages, and the local component can use the IPC functionality to act on messages without requiring an MQTT connection.

Client device best practices

The client device capabilities provided by the new Moquette MQTT broker, MQTT bridge, and Client device auth components allows for secure and flexible communication between an MQTT device (client device) and local MQTT broker, AWS IoT Core topics, and with custom components via IPC. Here are some best practices when implementing client devices.

Client devices require access public AWS IoT Greengrass endpoints

In order for a client device to connect to the AWS IoT Greengrass Core, it will need to know the local IP address, hostname, and the certificate authority that signed the server certificate used by the MQTT broker. To accomplish this, the client device will need to make an authenticated call to the public endpoint which will return the needed information.

This discovery process uses the client device’s X.509 certificate and private key to authenticate when making an API call. While this can be performed through a regular RESTful call and processing the results, the AWS IoT Device SDKs, Mobile SDKs, and AWS IoT Device Client have the AWS IoT Greengrass Core discovery functionality built in. Using these SDKs for client device provide not only general MQTT capabilities but also the full discovery process. If you are using your own MQTT libraries, please refer to the Interact with local IoT devices documentation for full details.

Use the IPC for component communication where possible

While client devices with X.509 certificates can use MQTT for their communication, AWS IoT Greengrass components should use the IPC method for all of their publish and subscribe operations. The IPC supports the same fine-grained security controls on a per-component basis and doesn’t require the overhead of an MQTT protocol stack. Finally, the IPC support interoperability with both the local Moquette MQTT broker via the MQTT bridge component and publish/subscribe IPC service, and to the AWS IoT Core via the AWS IoT Core MQTT messaging IPC service.

The only time you would use the client device pattern is if your components are written in a programming language where an AWS IoT device SDK exists. In that case you can have a component act as a client device, but you will have to manage the creation of a thing and certificate in AWS IoT Core.

Model and test all topic interactions between the Moquette MQTT broker, IPC, and AWS IoT Core

When creating client devices interaction between the Moquette MQTT broker and other services, it is recommended to map out the message flows. By mapping publish to subscribe operations, you can visualize and refine interactions. This can also help to ensure that message loops between the Moquette MQTT broker and AWS IoT Core do not take place.

Once you have mapped the interactions, a series of test cases for all potential interactions can help verify the Client devices auth component permissions are properly setup.

Summary

In this blog post I described the new client device functionality of AWS IoT Greengrass and demonstrated some example patterns of usage. The examples show how client devices can communication locally with each other or via AWS IoT Greengrass Core using the Moquette MQTT broker component. I also showed how local messages between the Moquette MQTT broker, AWS IoT Core, and components using the IPC can be routed and acted upon.

With this overview, you should now be able to incorporate your existing AWS IoT things into new or existing AWS IoT Greengrass Version 2 deployments. For more details on how to interact with client devices and the components used, please refer to the documentation pages here:

To become familiar with other AWS IoT Greengrass features and capabilities, here are some additional links to documentation and workshops:

What’s new – Summary page of release notes describing new features and changes
AWS-provided components – All public components available for inclusion into an AWS IoT Greengrass deployment
Performing machine learning inference – A popular use-case for using AWS IoT Greengrass ML inference and how it works
AWS IoT Greengrass v2 workshop – Self-paced workshop that goes through setup and main features of AWS IoT Greengrass Version 2

The Internet of Things on AWS – Official Blog