The Internet of Things on AWS – Official Blog

Archive AWS IoT Device Shadows in Amazon OpenSearch Service

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

In AWS IoT, you can create a device shadow (sometimes referred to as a thing shadow) that will be used as a communication layer between your mobile/cloud application and your devices connected to AWS IoT. The shadow is a persistent, virtual representation of your devices. Because it always has a point-in-time view of the state of your device, it’s easy to write applications that interact with your devices through device shadows.

When you update the shadow of your device, the Thing Shadows service will typically publish two messages:

/update/accepted or /update/rejected
This message shows whether the update you sent was accepted by the service. It will show which fields were updated and include metadata containing the last updated timestamps of those fields.

/update/delta
This message is published whenever the Thing Shadows service detects a difference between the desired and reported sections of the thing shadow.

We’ve now introduced a new message for shadow updates that will be published for every update:

/update/documents
This message will contain the entire shadow document before the service processes the request and the entire shadow document after the request is processed.

The new documents messages, coupled with AWS IoT integration in Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), make it easy to forward your shadows to an Amazon OpenSearch Service cluster and then use Kibana to visualize the historical shadow data.

In this post, I’ll walk you through sending data to your shadow, indexing the shadow in Amazon OpenSearch Service, and visualizing it in Kibana.

Create an Amazon OpenSearch Service Domain

Let’s start with creating an Amazon OpenSearch Service domain to store the historical shadow state.

In the Amazon OpenSearch Service console, create a domain with name delivery-fleet.

In the domain creation wizard, you will be asked to set the domain access policy. You need to specify an access policy that:

Allows AWS IoT to put data into the Amazon OpenSearch Service domain.
Allows intended clients (such as your desktop) to query data from the Amazon OpenSearch Service domain.
The access policy you choose should look similar to the following example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:role/iot-es-action-role"
      },
      "Action": "es:ESHttpPut",
      "Resource": "arn:aws:es:us-east-1:xxxxxxxxxxxx:domain/delivery-fleet/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:xxxxxxxxxxxx:domain/delivery-fleet/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "xxx.xxx.xxx.xxx",
            "xxx.xxx.xxx.xxx"
          ]
        }
      }
    }
  ]
}

Specify the public IP addresses or address ranges of your intended clients (such as your desktop) in the aws:SourceIp list. To find the public IP address of a client, on the client machine, go to https://www.google.com/#q=what+is+my+public+ip+address.

Create the Amazon OpenSearch Service domain. It will take a few minutes. When Active is displayed for  Domain status, make a note of the domain endpoint. The endpoint will look similar to search-delivery-fleet-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com.

Next, create an index in the Amazon OpenSearch Service domain through an HTTP request. You can use an HTTP client such as curl or DHC. In Amazon OpenSearch Service, an index is the top-level logical structure where you store and index your data. For this example, create an index with the name trucks to store the delivery truck data. As you create the index, specify a mapping in the HTTP request body to help Amazon OpenSearch Service correctly interpret geo-location and time from the sample data.

Your HTTP POST request should look similar to the following example. You should get an HTTP 200 in response.

curl -XPOST https://search-delivery-fleet-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com/trucks -d '{
 "mappings": {
    "truck": {
      "properties": {
         "timestampMillis": {
          "type": "long",
          "copy_to": "datetime"
        },
        "datetime": {
          "type": "date",
          "store": true
        },
        "state": {
          "properties": {
            "reported": {
              "properties": {
                "location": {
                  "type": "geo_point"
                }
              }
            }
          }
        }
      }
    }
  }
}'

Configure an AWS IoT Rule to Route Data to Amazon OpenSearch Service

Now that your Elasticsearch index is ready to receive data, configure an AWS IoT rule that can route the inbound data from connected trucks to the Elasticsearch index. You can use the AWS IoT console or the AWS CLI to create a rule. Your AWS IoT rule should look like the following example:

{
"sql": "SELECT current.state AS state, cast(timestamp as number) * 1000 AS timestampMillis FROM '$aws/things/truck42/shadow/update/documents'",

"actions": [

{

"elasticsearch": {

"roleArn": "arn:aws:iam::xxxxxxxxxxxx:role/iot-es-action-role",

"endpoint": "https://search-delivery-fleet- xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com",

"index": "trucks",

"type": "truck",

"id": "${newuuid()}"

}

}

],

"ruleDisabled": false,

"awsIotSqlVersion": "2016-03-23-beta"

}

The example rule forwards the entire state document under the current subtree from the documents topic for the device shadow named truck42 to the Elasticsearch index trucks.

The AWS IoT service must assume an AWS IAM role in your AWS account to obtain the permissions required to insert data into your Amazon OpenSearch Service domain. In particular, the IAM role must be allowed to call the es:ESHttpPut action.

Simulate an Internet-Connected Truck Sending Data to Your AWS IoT Device Shadow

You are now ready to send data to the AWS IoT device shadow where it will be indexed into Amazon OpenSearch Service.

Create an AWS Lambda function using this sample code to simulate an Internet-connected truck sending location and performance metrics to AWS IoT. The code is designed to update your device shadow in AWS IoT at periodic intervals with the following state:

{
  "state": {
    "reported": {
      "nms": 1412638168724,
      "location": "39.09972,-94.57853",
      "geoJSON": {
        "type": "Point",
        "coordinates": [
          -94.57853,
          39.09972
        ]
      },
      "pressure": 111,
      "engine_temperature": 213,
      "cargo_temperature": 41,
      "rpm": 2216,
      "speed": 18,
      "battery": 12.3
    }
  }
}

AWS IoT generates an account-specific endpoint for your devices to send and receive data to and from your AWS account. When you create the Lambda function, you will need to update the code with your account-specific AWS IoT endpoint. To find this endpoint, use the aws iot describe-endpoint AWS CLI command.

Make sure that the Lambda execution role has permission to invoke the iot:UpdateThingShadow action. You can use the AWS IAM console to edit the role’s policy and grant this permission.

Lastly, increase the execution timeout for the Lambda function from the default of 3 seconds to 5 minutes. This allows the function to add a delay between sending several consecutive messages. You can configure the timeout under Advanced Settings. (Later, when you trigger the function through the console, you will get a warning, which you can safely ignore: “We are unable to display results and logs for invocations that take longer than 60 seconds. You can view the results and logs for the function in Amazon CloudWatch once the function completes executing.”)

You are now just one click away from running the simulation. Before you trigger the simulation, optionally configure one or both of the following data flow debugging tools:

First, use an MQTT client to confirm delivery of truck data as acknowledged by the Thing Shadows service. There is a browser-based MQTT client available on the upper-right corner of the AWS IoT console. Connect and subscribe to topic $aws/things/truck42/shadow/update/#. This will allow you to view the messages published by the device shadow as it sees the updates come into the shadow. For more information, see Using the MQTT Client in the AWS IoT Developer Guide.

Second, configure AWS IoT to send logs to Amazon CloudWatch Logs. Viewing the logs there is useful, in case you need to debug issues like authentication or rule execution failures. For more information, see Setting up Cloudwatch Logs in the AWS IoT Developer Guide.

In the AWS Lambda console, choose the Test button to run the truck simulator Lambda function. You should see messages coming from all destinations, including the browser-based MQTT client and the Amazon OpenSearch Service domain.

Explore and Visualize the Shadow Data in Kibana

You are now ready to explore the connected truck shadow data. First, confirm that your Amazon OpenSearch Service domain has received and indexed the truck data by querying Amazon OpenSearch Service through an HTTP GET request. Use an HTTP client such as curl or DHC. Your HTTP GET request should look like the following example and you should get an HTTP 200 with a non-zero number of hits in the response body:

curl -i -X GET 'https://search-delivery-fleet-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com/trucks/_search'

The next step is to start using Kibana. In the Amazon OpenSearch Service console, look up and choose the Kibana endpoint for your Amazon OpenSearch Service domain. It will look like: search-delivery-fleet-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com/_plugin/kibana/.

On the Configure an index pattern page, for Index name or pattern, type trucks and under Time-field name,choose datetime. Kibana will then list the fields found under the trucks index:

Go to the Discover tab to start exploring the data. By default, Kibana sets the timeframe to Last 15 minutes. You may need to increase it to Last 1 hour or some other appropriate timeframe to include the time when the truck shadow simulator Lambda function published the data. Kibana will load the data as shown in the following example:


Next, use the Visualize tab for more interesting visualizations. The following example is a map that shows the maximum speed of the truck against a geohash of the location data in the AWS IoT device shadow for the truck:


This walkthrough showed you how to archive an AWS IoT device shadow in an Amazon OpenSearch Service domain. We hope you found it useful. Try it out, and feel free to leave your feedback in the comments.