The Internet of Things on AWS – Official Blog

Harnessing the power of AWS IoT rules with substitution templates

AWS IoT Core is a managed service that helps you to securely connect billions of Internet of Things (IoT) devices to the AWS cloud. The AWS IoT rules engine is a component of AWS IoT Core and provides SQL-like capabilities to filter, transform, and decode your IoT device data. You can use AWS IoT rules to route data to more than 20 AWS services and HTTP endpoints using AWS IoT rule actions. Substitution templates are a capability in IoT rules that augments the JSON data returned when a rule is triggered and AWS IoT performs an action. This blog post explores how AWS IoT rule actions with substitution templates unlock simpler, more powerful IoT architectures. You’ll learn proven ways to cut costs and enhance scalability. Through practical examples of message routing and load balancing, smarter, more efficient IoT solutions.

Understanding the fundamental components

Each AWS IoT rule is built upon three fundamental components: a SQL-like statement that handles message filtering and transformation, one or more IoT rule actions that run and route data to different AWS and third party services, and optional functions that can be utilized in both the SQL statement and rule actions.

The following is an example of an AWS IoT rule and its components.

{
   "sql": "SELECT *, get_mqtt_property(name) FROM 'devices/+/telemetry'", 
   "actions":[
    {
      "s3":{  
        "roleArn": "arn:aws:iam::123456789012:role/aws_iot_s3",
        "bucketname": "MyBucket",
        "key" : "MyS3Key"
      }
    }
   ]
}

The SQL statement serves as the gateway for rule processing and determines which MQTT messages should be handled based on specific topic patterns and conditions. The rule employs a SQL-like and supports SELECT, FROM, and WHERE clauses (for more information, see AWS IoT SQL reference). Within this structure, the FROM clause defines the MQTT topic filter, and the SELECT and WHERE clauses specify which data elements should be extracted or transformed from the incoming message.

Functions are essential to the SQL statement and IoT rule actions. AWS IoT rules provide an extensive collection of internal functions designed to convert data types, manipulate strings, perform mathematical calculations, handle timestamps, and much more. Additionally, AWS IoT rules provide a set of external functions that help you to retrieve data from AWS services (such as, Amazon DynamoDB, AWS Lambda, Amazon Secrets Manager, and AWS IoT Device Shadow) and embed that data in your message payload. These functions support sophisticated data transformations directly within the rule processing pipeline and eliminates the need for external processing.

Rule actions determine the destination and handling of processed data. AWS IoT rules support a library of built-in rule actions that can transmit data to AWS services, like AWS Lambda, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, and Amazon Simple Queue Service (Amazon SQS). These rule actions can also transmit data to third-party services like Apache Kafka. Each rule action can be configured with specific parameters that govern how the data should be delivered or processed by the target service.

Substitution templates: The hidden gem

You can implement functions within the AWS IoT rule SELECT and WHERE statements to transform and prepare message payloads. If you apply this approach too frequently, however, you might overlook the powerful option to use substitution templates and perform transformations directly within the IoT rule action.

Substitution templates support dynamically inserted values and rule functions into the rule action’s JSON using the ${expression} syntax. These templates support many SQL statement functions, such as timestamp manipulation, encoding/decoding operations, string processing, and topic extraction. When you utilize substitution templates within AWS IoT rule actions, you can implement sophisticated routing that significantly reduces the complexity in other architectural layers, resulting in more efficient and maintainable AWS IoT solutions.

Real-world implementation patterns

Let’s dive into some practical examples that show the versatility and power of using substitution templates in AWS IoT rules actions. These examples will demonstrate how this feature can simplify your IoT data processing pipelines and unlock new capabilities in your IoT applications.

Example 1: Conditional message distribution using AWS IoT registry attributes

Consider a common IoT scenario where a platform distributes device messages to different business partners, and each partner has their own message processing SQS queue. Different partners own each device in the fleet and their relationship is maintained in the registry as a thing attribute called partnerId.

The traditional approach includes the following:

  • Option 1 – Maintain partner routing logic on the device. Multiple AWS IoT rules rely on WHERE conditions to input payload:
    • Requires devices to know their partner’s ID.
    • Increases device complexity and maintenance.
    • Creates security concerns with exposing partner identifiers.
    • Makes partner changes difficult to manage.
  • Option 2 – Employ an intermediary Lambda function to retrieve the partner ID values associated with devices from the AWS IoT registry and subsequently propagate the message to the partner specific SQS queue:
    • Adds unnecessary compute and registry query costs.
    • Potentially increases message latency.
    • Creates additional points of failure.
    • Requires maintenance of routing logic.
    • May face Lambda concurrency limits.

Here’s a more elegant solution and process that uses substitution templates and the new AWS IoT propagating attributes feature:

  • Insert the Partner IDs as attributes in the AWS IoT registry
  • Use the propagating attributes feature to enrich your MQTTv5 user property and dynamically construct the Amazon SQS queue URL using the device’s partnerId. See the following example:
{
    "ruleArn": "arn:aws:iot:us-east-1:123456789012:rule/partnerMessageRouting",
    "rule": {
        "ruleName": "partnerMessageRouting",
        "sql": "SELECT * FROM 'devices/+/telemetry'",
        "actions": [{
            "sqs": {
                "queueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/partner-queue-${get(get_user_properties('partnerId'),0}}",
                "roleArn": "arn:aws:iam::123456789012:role/service-role/iotRuleSQSRole",
                "useBase64": false
            }
        }],
        "ruleDisabled": false,
        "awsIotSqlVersion": "2016-03-23"
    }
}

Using this solution, a device with partnerId=”partner123″ publishes a message. The message is automatically routed to the “partner-queue-partner123” SQS queue.

Benefits of this solution:

Using the substitution template significantly simplifies the architecture and provides a scalable and maintainable solution for partner-specific message distribution. The solution,

  • Eliminates the need for additional compute resources.
  • Provides immediate routing without added latency.
  • Simplifies partner relationship management through updates in the AWS IoT thing registry. For example, introducing new partners, can be updated by modifying the registry attributes. This update wouldn’t require any updates or changes to the devices or the routing logic.
  • Maintains security by not exposing queue information to devices.

Example 2: Intelligent load balancing with Amazon Kinesis Data Firehose

Consider a scenario where millions of devices publish telemetry data to the same topic. There is also a need to distribute this high-volume data across multiple Amazon Data Firehose streams to avoid throttling issues when buffering the data to Amazon S3.

The traditional approach includes the following:

  • Device-side load balancing:
    • Implement configuration management to provide different stream IDs across the devices.
    • Require the devices to include stream targeting in their messages.
    • Create multiple AWS IoT rules to match the specific stream IDs.
  • AWS Lambda-based routing:
    • Deploy a Lambda function to distribute messages across streams.
    • Implement custom load balancing logic.

Traditional approaches exhibit similar negative impacts as outlined in the preceding example (maintenance overhead, security vulnerabilities, device complexity, additional costs, increased latency, and failure points). Furthermore, they present specific challenges in high-volume scenarios, such as heightened risk of throttling and complex streams management.

By leveraging AWS IoT rule substitution templates, you can implement a streamlined, serverless load balancing solution that dynamically assigns messages to different Firehose delivery streams by:

  1. Generate a random number between 0-100000 using rand()*100000.
  2. Convert (casting) this random number to an integer.
  3. Use modulo operation (mod) to get the remainder when divided by 8.
  4. Append this remainder (0-7) to the base name “firehose_stream_”.

The result is that messages are randomly distributed across eight different Amazon Data Firehose streams (firehose_stream_0 through firehose_stream_7). See the following example:

{ 
  "ruleArn": 
    "arn:aws:iot:us-east-1:123456789012:rule/testFirehoseBalancing", 
  "rule": { 
    "ruleName": "testFirehoseBalancing", 
    "sql": "SELECT * FROM 'devices/+/telemetry'", 
    "description": "", 
    "createdAt": "2025-04-11T11:09:02+00:00", 
    "actions": [ 
        { "firehose": { 
            "roleArn": "arn:aws:iam::123456789012:role/service-role/firebaseDistributionRoleDemo", 
            "deliveryStreamName": "firehose_stream_${mod(cast((rand()*100000) as Int),8)}", 
            "separator": ",",
            "batchMode": false 
        } 
     } 
    ], 
  "ruleDisabled": false, 
  "awsIotSqlVersion": "2016-03-23" 
  }
}

Benefits of this solution:

This flexible load balancing pattern helps to handle high message volumes by spreading the load across multiple streams. The primary advantage of this approach lies in its scalability. By modifying the modulo function (which determines the remainder of a division, for instance, 5 mod 3 = 2), the dividend (currently set to 8) can be adjusted to correspond with the desired number of streams. For example:

  • Change to mod(…, 4) for distribution across 4 streams.
  • Change to mod(…, 16) for distribution across 16 streams.

Using this template makes it easy to scale your architecture up or down without changing the core logic of the rule.

Example 3: Use CASE statements in substitution templates to build a conditional routing logic

Consider a scenario where you need to route your IoT device data, depending on the specific device, either to a production-based or to a Development/Testing (Dev/Test) Lambda function.

The traditional approach includes the following:

  • Device-side load balancing:
    • Implement configuration management to provide different environment IDs across the devices.
    • Require the devices to include an environment IDs in their messages.
    • Create multiple AWS IoT rules to match the specific environment IDs.
  • AWS Lambda-based routing:
    • Deploy a Lambda function to distribute messages across the different environment AWS Lambda functions after a check against the AWS IoT registry (or an alternative database).

Traditional approaches exhibit the same negative impacts as outlined in the preceding examples.

Here’s a more elegant solution and process that uses substitution templates and the new AWS IoT propagating attributes feature:

  • Associate the environment IDs as attributes for all devices in the AWS IoT Registry
  • Use the propagating attributes feature to enrich your MQTTv5 user property
  • Utilize the propagated property to dynamically construct the AWS Lambda function ARN within a CASE statement embedded within the AWS IoT Rule action definition.

See the following example:

{ 
  "ruleArn": 
    "arn:aws:iot:us-east-1:123456789012:rule/ConditionalActions", 
  "rule": { 
    "ruleName": "testLambdaConditions", 
    "sql": "SELECT * FROM 'devices/+/telemetry'", 
    "description": "", 
    "createdAt": "2025-04-11T11:09:02+00:00", 
    "actions": [ 
        { "lambda": { 
            "functionArn": 
                "arn:aws:lambda:us-east-1:123456789012:function:${CASE get(get_user_properties('environment'),0) 
                    WHEN \"PROD\" THEN \"message_handler_PROD\" 
                    WHEN \"DEV\" THEN \"message_handler_DEV\" 
                    WHEN NULL THEN \"message_handler_PROD\" 
                    ELSE \"message_handler_PROD\" END }",  
        } 
     } 
  ], 
  "ruleDisabled": false, 
  "awsIotSqlVersion": "2016-03-23" 
 }
}

Benefits of this solution:

Using the substitution template significantly simplifies the architecture and provides a scalable and maintainable solution for partner-specific message distribution. The solution,

  • Removes the requirement to define separate IoT rule and IoT rule actions for each condition.
  • Helps you reduce the cost of using IoT rules and IoT rule actions.

Conclusion

This blog post explored how substitution templates for AWS IoT rules can transform complex IoT architectures into elegant and efficient solutions. The examples demonstrated that substitution templates are more than just a feature – they’re a powerful architectural tool that leverages AWS IoT capabilities to efficiently solve complex challenges without introducing additional complexity or cost. Substitution templates provide a serverless, scalable approach that eliminates the need for additional compute resources or complex client-side logic. This approach not only reduces operational overhead but also provides immediate cost benefits by removing unnecessary compute resources and simplifying the overall architecture.

The next time you find yourself designing AWS IoT message routing patterns or facing scaling challenges, consider how a substitution template might offer a simpler and more efficient solution. By leveraging these powerful AWS IoT features, you can create more maintainable, cost-effective, and scalable IoT solutions that truly serve your business needs.

Remember: The simplest solution is often the most elegant one. With AWS IoT rule substitution templates, that simplicity comes built in.

About the Authors

Andrea Sichel is a Principal Specialist IoT Solutions Architect at Amazon Web Services, where he helps customers navigate their cloud adoption journey in the IoT space. Driven by curiosity and a customer-first mindset, he works on developing innovative solutions while staying at the forefront of cloud technology. Andrea enjoys tackling complex challenges and helping organizations think big about their IoT transformations. Outside of work, Andrea coaches his son’s soccer team and pursues his passion for photography. When not behind the camera or on the soccer field, you can find him swimming laps to stay active and maintain a healthy work-life balance.

Avinash Upadhyaya is Senior Product Manager for AWS IoT Core where he is responsible to define product strategy, roadmap prioritization, pricing, and a go-to-market strategy for features within the AWS IoT service.