Introduction

IoT applications are synonymous with the Publish / Subscribe (PubSub) model where distributed services communicate via event triggered messages. The PubSub model offers flexibility in design and is well suited to event-driven distributed IoT systems. However, this flexibility puts many design decisions in the hands of the developer that creates dependencies across systems, services and teams. In this post, you will learn how to accelerate and simplify the development of distributed IoT PubSub applications deployed and managed by AWS IoT Greengrass using the AWS IoT Greengrass PubSub SDK for Python.

Solution Overview

The AWS IoT Greengrass PubSub SDK for Python (referred to here as the SDK) provides a highly opinionated IoT PubSub application architecture delivered as a Python library for AWS Greengrass V2 Components. The SDK abstracts the AWS IoT Greengrass PubSub functionality and defines a data-driven message format to automatically route AWS IoT Greengrass IPC / MQTT messages to user defined application call-backs. In doing so, the SDK delivers a No/Low-Code messaging service that decouples the PubSub topic schema from the desired interface functionality.

The following diagram provides a high level overview of the AWS IoT Greengrass PubSub SDK for Python architecture:

In a typical IoT PubSub service, the message payload format and the topic schema are tightly coupled to the APIs functionality. Using this approach creates complex and difficult to manage upstream dependencies. A final challenge is the repeated development of boilerplate code to manage PubSub message processing and routing.

The following diagram describes a typical distributed IoT application service interface with tight integrations and dependencies between message formats and topics and the individual service functions.

The AWS IoT Greengrass PubSub SDK for Python solves these key dependencies and challenges by providing:

1. Defined Message Format: The SDK provides a defined but extendible message format with control fields that support data-driven message routing to decouple PubSub topics from interface functionality.
2. Simplified Topic Schema: The SDK defines a simple but extendible topic schema consisting of just a single ingress and egress topic made possible by data-driven message routing.
3. Messaging Service Boilerplate Code: The SDK abstracts the AWS IoT Greengrass IPC and MQTT PubSub functionality to process and route PubSub messages to user-defined application call-backs.

The below diagram displays a similar service interface when using the AWS IoT Greengrass PubSub SDK with message formats and topic schema no longer coupled to service functionality. This is made possible by data driven message routing and removes the upstream dependency and change management on these public facing service attributes.

PubSub Message Process and Routing

Message Format

The SDK message routing is controlled by the route field in the prescribed message format shown below:

{
  "sdk_version": "0.1.0",
  "message_id": "20220403170948930231",
  "status": 200,
  "route": "MyPubSubMessageHandler.pubsub_message_callback_function",
  "message": {
    "my-message-param01": "param01",
    "my-message-param02": "param02"
  }
}

The route field describes a route target that is a (Plain Old Python) call-back function in your application logic using Class.Function namespace convention.

1. sdk_version: Provides message version control across distributed systems. By default, the SDK will reject messages with a different major version.
2. message_id: This is typically a timestamp but can be any user defined string. Its is used to track messages across asynchronous Request / Response patterns.
3. status:  Status of the message request, response or update. When used with message_id, this allows asynchronous Request / Response messages to report a given status without the need for success / failure topics and message formats.
4. route: As described, is used by the SDK to route PubSub messages to user defined application call-backs in Class.Function namespace convention.
5. message: User defined payload field for SDK formatted messages. The SDK will forward this unchanged to the user defined callbacks along with the other described fields.

The below diagram describes in more detail how the SDK defined message format is used to route messages from the wire to the user defined application call-backs:

1. An AWS IoT Greengrass component deployed with the SDK receives a well-formatted message with a valid route field of SystemMessageHandler.health_check.
2. The SDK processes the message, checks for validity, SDK version and determines if the route matches a user defined application call-back and routes the message accordingly.
3. The PubSub message is forwarded to the health_check function with all SDK message fields parameterized for processing by user application logic.
4. Here we assume the health_check function creates a well formatted response with the system health in the payload, a new valid route and reflects the message_id for upstream systems.
5. SDK receives the message publish request from the health_check function and forwards the message to the appropriate PubSub topic.

This data driven message routing allows you to update service interface functionality with only minor changes needed. For example, to add the interface to support the described system health check request; just add the appropriate callback function in your application message handler and then advertise the given route to upstream services. There is no change to PubSub topic schema, very minimal code changes and is non-breaking and backwards compatible with existing interface functionality.

Topic Schema

Due to the message-based routing fields, the SDK base topic schema can consist of just two topics, the Ingress and Egress topics:

• Ingress Topic: BASE_TOPIC/AWS_IOT_THING_NAME/ingress
• Egress Topic: BASE_TOPIC/AWS_IOT_THING_NAME/egress

Where BASE_TOPIC is a user defined parameterized field provided when initializing each SDK based AWS Greengrass Component. It is typically set to the component name.

This simple topic schema is extensible via parameter injection from the component recipe (JSON) file thus allowing topic schema changes and custom subscriptions without the need for code changes. In this way, the SDK manages the interface contract between services so you as the developer can focus exclusively on delivering quality application logic.

Deploying the AWS IoT Greengrass PubSub SDK

The easiest way to get started and deploy the AWS IoT Greengrass PubSub SDK is to replicate the sample component provided in the SDK GitHub repository and deploy via the AWS IoT Greengrass Development Kit (GDK). There are two resources that describe this process in detail:

1. SDK Sample Directory: The SDK provides a sample AWS IoT Greengrass component with the SDK installed. Follow the guide provided in the Samples directory to deploy a single component and to send a simple MQTT PubSub Request / Response message.
2. Build a Distributed IoT Application with the AWS IoT Greengrass PubSub SDK workshop: This workshop builds a Smart Factory distributed IoT PubSub Application with integrations to Amazon SQS using the SDK.

Further Reading

The AWS IoT Greengrass PubSub SDK comes with a detailed developer’s guide and API docs provided in the below links:

• Developer Guide
• API Docs
• PyPi AWS Greengrass PubSub SDK Library

For further reading on AWS IoT Greengrass and AWS IoT more broadly see the following links:

• AWS Greengrass V2 Developer Guide
• AWS IoT Core Documentation
• AWS Dev Blog

Conclusion

In this post, I discussed the common challenges developers face when designing distributed IoT PubSub applications. I introduced the AWS IoT Greengrass PubSub SDK and explained how using this solution can address many of these challenges and dependencies. To get started building, we recommend you try the Build a Distributed IoT Application with the AWS IoT Greengrass PubSub SDK workshop.

About the author

Dean Colcott is a Snr IoT and Robotics Specialist Solution Architect at Amazon Web Services (AWS) with areas of depth in distributed applications and full stack development, video analytics and computer vision and Industrial IoT and Enterprise data platforms. Dean’s focus is to bring relevance, value and insight from IoT and Industrial data and data generating physical assets within the context of the Enterprise data strategy.