The Internet of Things on AWS – Official Blog

Getting Aggregate Information of Devices with AWS IoT Device Management Fleet Indexing

AWS IoT Device Management Fleet Indexing has launched three APIs for getting aggregate information of devices: GetStatistics, GetPercentiles and GetCardinality. With these APIs, you can quickly answer questions like: What’s the percentage of the devices connecting to AWS IoT? What’s the maximum, minimum and average of device battery level? How many OS versions are present?

In this blog post, we will walk you through a data center temperature monitoring simulation to understand how to use Fleet Indexing APIs to get aggregate information of the devices. A data center may have many sensors monitoring server temperatures. You will learn how to use GetStatistics to get the min/max/avg temperature; use GetPercentiles to get the 90th percentile of temperature; use GetCardinality to get the number of server racks with abnormal temperatures.

Set Up

Ensure you have the latest version of the AWS CLI installed.

Assuming you are operating a data center with 10 temperature sensors connected to AWS IoT, distributed among 6 server racks. Some sensors are reporting abnormal temperatures. Let’s use the script below to create 10 things representing the sensors. Each thing will have three attributes:

  • temperature: Number type – temperature value in Fahrenheit. Note that using thing attribute here is for simplicity of demonstration only. For best practices, please use AWS IoT Device Shadows for frequently changing fields.
  • rackId: String type – id of the server rack where the sensor belongs. Ranging from Rack1 to Rack6.
  • stateNormal: Boolean type – indicating if the temperature value is within normal range: [60 to 80]. Rack5 and Rack6 are in abnormal state.
# Bash script. If in other shells, type `bash` before running

Temperatures=(70 71 72 73 74 75 47 97 98 99)
Racks=(Rack1 Rack1 Rack2 Rack2 Rack3 Rack4 Rack5 Rack6 Rack6 Rack6)
IsNormal=(true true true true true true false false false false)

for ((i=0; i<10 ; i++))
do
  thing=$(aws iot create-thing --thing-name "TempSensor$i" --attribute-payload attributes="{temperature=${Temperatures[i]},rackId=${Racks[i]},stateNormal=${IsNormal[i]}}")

  aws iot describe-thing --thing-name "TempSensor$i"
done

To get aggregate information of these sensors, you need to enable Fleet Indexing and define what fields to aggregate with the UpdateIndexingConfiguration API. In the sample code below, thing-indexing-configuration defines how things are indexed.

  • thingIndexingMode: One of “OFF”|”REGISTRY”|”REGISTRY_AND_SHADOW”. Here REGISTRY means Fleet Indexing will only index the registry data of a thing.
  • customFields: a list of field name and type pairs. In order to perform aggregation queries on a field, we have to include the field name and type, when setting up the indexing configuration. However, this does not apply to AWS IoT managed fields such as thingName, thingId and registry.version. Managed fields support aggregation queries by default.
aws iot update-indexing-configuration --thing-indexing-configuration "thingIndexingMode=REGISTRY,customFields=[{name=attributes.temperature,type=Number},{name=attributes.rackId,type=String},{name=attributes.stateNormal,type=Boolean}]"

After turning on indexing, Fleet Indexing will create an index called AWS_Things. The index will first be in the BUILDING or REBUILDING state. After the existing things are indexed, the index status will flip to ACTIVE and be ready for queries. With 10 things, your index should be ACTIVE within seconds to a minute. You can check the index status with the following command

aws iot describe-index --index-name "AWS_Things"

Result:
{
    "indexName": "AWS_Things",
    "indexStatus": "ACTIVE",
    "schema": "REGISTRY"
}

Perform Aggregation Queries

Once the index is ACTIVE, you can use GetStatistics API to monitor the statistics of all temperature sensors.

  • aggregation-field specifies which field to aggregate. Note, only custom fields defined with UpdateIndexingConfiguration or AWS IoT managed fields can be aggregated.
  • query-string defines a filter on things matching the query criteria.
aws iot get-statistics --aggregation-field "attributes.temperature" --query-string "thingName:TempSensor*"

Result:
{
    "statistics": {
        "count": 10,
        "average": 77.6,
        "sum": 776.0,
        "minimum": 47.0,
        "maximum": 99.0,
        "sumOfSquares": 62578.0,
        "variance": 236.04000000000013,
        "stdDeviation": 15.36359332968691
    }
}

The max and min temperatures are 99 and 47, which indicates some sensors are reading anomalies. To better understand the distribution, you can leverage GetPercentiles. This feature returns a list of probability and non-exceedance field value pairs. For example, in the following result you have {percent:90.0, value:98.1}. This means 90% of the temperature values are below 98.1. The input percentages defines a list of percentiles to return. Notice GetPercentiles gives approximation results. Larger aggregation set leads to more accurate estimations.

aws iot get-percentiles --aggregation-field "attributes.temperature" --query-string "thingName:TempSensor*" --percents 10 25 50 75 90

Result:
{
    "percentiles": [
        {
            "percent": 10.0,
            "value": 67.7
        },
        {
            "percent": 25.0,
            "value": 71.25
        },
        {
            "percent": 50.0,
            "value": 73.5
        },
        {
            "percent": 75.0,
            "value": 91.75
        },
        {
            "percent": 90.0,
            "value": 98.1
        }
    ]
}

Combining the two responses above, you can conclude a few sensors are detecting anomalies. Among them, most are reporting high temperatures. With GetCardinality, you can quickly check if the temperature problem is happening at one location (in one or two racks) or across the entire data center (all racks). Same as GetPercentiles, the results are estimations and accuracy improves over the scale. The result below means only two racks are having abnormal temperatures.

aws iot get-cardinality --aggregation-field "attributes.rackId" --query-string "thingName:TempSensor* AND attributes.stateNormal:false"

Result:
{
    "cardinality": 2
}

Among all the racks, let’s assume you know Rack6 has more servers than others. You can quickly look at the status of Rack6 using GetStatistics. From the result below, you can confirm Rack6 is too hot as the average temperature is 98.0.

aws iot get-statistics --aggregation-field "attributes.temperature" --query-string "thingName:TempSensor* AND attributes.rackId:Rack6"

Result:
{
    "statistics": {
        "count": 3,
        "average": 98.0,
        "sum": 294.0,
        "minimum": 97.0,
        "maximum": 99.0,
        "sumOfSquares": 28814.0,
        "variance": 0.6666666666666666,
        "stdDeviation": 0.816496580927726
    }
}

After mitigating the problem, you can investigate the low temperature value of 47 with another Fleet Indexing API: SearchIndex. This API returns all the things satisfying the query string. Based on the result below, you know the low value is from TempSensor6 and actions are needed at Rack5.

aws iot search-index --query-string "thingName:TempSensor* AND attributes.temperature<60"

Result:
{
    "things": [
        {
            "thingName": "TempSensor6",
            "thingId": "45b45d4e-ad95-495f-828e-47df5454ab7c",
            "attributes": {
                "rackId": "Rack5",
                "stateNormal": "false",
                "temperature": "47"
            }
        }
    ]
}

Clean Up

If you no longer need the resources in the example, the following script can help to delete all the things.

for ((i=0; i<10; i++))
do
  aws iot delete-thing --thing-name "TempSensor$i"
done

It is recommended to keep Fleet Indexing enabled so the devices are always ready for queries. Optionally, you can turn thing indexing off with the following command.

aws iot update-indexing-configuration --thing-indexing-configuration thingIndexingMode=OFF

Conclusion

We have demonstrated how to use AWS IoT Device Management Fleet Indexing APIs to perform aggregation queries on devices. These APIs also work for thing group aggregations. The prerequisite is to enable Fleet Indexing and define the fields you want to aggregate with the UpdateIndexingConfiguration API.

While performing aggregation queries, you use

  • query-string to filter in the things matching the query criteria.
  • aggregation-field to specify what field to aggregate. Only custom fields defined with UpdateIndexingConfiguration or AWS IoT managed fields can be aggregated.

Given a query and a field

  • GetStatistics returns basic statistics (count/avg/sum/min/max etc) of the field value.
  • GetPercentiles returns an estimated list of percentile, field value pairs. Each pair indicates the percentile probability of a thing field value not exceeding the given field value.
  • GetCardinality returns an estimated count of unique field values.

Learn More

Fleet Indexing is a feature within AWS IoT Device Management. Apart from querying for aggregate data, there is much more that AWS IoT Device Management offers to register, organize, monitor, and remotely manage connected devices at scale.

AWS IoT Fleet Indexing Documentation
https://docs.aws.amazon.com/iot/latest/developerguide/iot-indexing

AWS IoT Device Management
https://aws.amazon.com/iot-device-management

AWS IoT Device Management Features
https://aws.amazon.com/iot-device-management/features