Ingesting and managing data from industrial equipment with AWS IoT SiteWise

AWS IoT SiteWise is a managed service that simplifies collecting, organizing, and analyzing industrial equipment data at scale. With AWS IoT SiteWise, customers can gather data reliably from multiple facilities, structure it, and make it accessible and understandable without developing additional software. With equipment data stored in AWS IoT SiteWise, customers can assess the performance of their industrial equipment remotely and across locations. This visibility across industrial facilities allows customers to streamline operations, as well as identify gaps in production and waste. In addition, AWS IoT SiteWise Edge runs on-premises, securely connecting to and reading data from equipment or local historian databases. On Nov 24th 2021, AWS IoT SiteWise announced three new enhancements that make it easier to ingest equipment data to the cloud. Prior to these enhancements, customers had to model their equipment before ingesting data to AWS IoT SiteWise. Now, Customers can ingest data into the cloud as soon as their equipment is connected to AWS IoT SiteWise. In the blog, we will cover details, benefits, and best practices for customers using AWS IoT SiteWise with disassociated data ingestion.

Improved asset modeling experience

Customers use AWS IoT SiteWise to build models of their physical operations that represent their assets, processes, and facilities, which will help them understand industrial data in the context of their equipment. Once asset models are created, customers can define an asset hierarchy to accurately represent relationships between devices and equipment within a single facility or across multiple facilities. Our customers tell us that modeling production operation and equipment could be a lengthy process, and it changes and evolves over time. Now with the disassociated data ingestion feature, all data streams will be ingested to the cloud without the prerequisite of associating them to assets. Customers can ingest data streams through an AWS IoT SiteWise Edge gateway, AWS IoT Core, or directly using AWS IoT SiteWise batch PUT API. It also gives you the flexibility to model your production operation after data ingestion. As your production environment evolves, you can adapt to changes and update data stream and asset association without any data loss. It leads to a virtuous cycle of asset modeling experience as illustrated in the virtuous cycle of asset modeling below.

Virtuous cycle of asset modeling

When you connect equipment to AWS IoT SiteWise, equipment data are represented as data streams. Each data stream represents a measurement that can be associated as a property for a virtual representation of equipment in AWS IoT SiteWise. An asset model is a virtual representation of a type of equipment and includes one or more properties that are measurements from the equipment. Using an asset model, you create an asset that represents a physical piece of equipment in your production operations. You can then define hierarchies of assets to organize virtual representations of equipment in your production environment. With the ability of disassociating and re-associating data streams to assets, you can continue to evolve asset modeling to represent the latest state of your production environment. For example, in discrete manufacturing, it’s common for a piece of equipment to be physically moved to another location. In this scenario, you can update Hierarchy definitions in asset model to reflect the latest change on the manufacturing floor. In addition, all data you collected is still retained with no loss during the reconfiguration.

Enabling disassociated data ingestion

Disassociated data ingestion is enabled by default for new AWS IoT SiteWise customers (AWS accounts that have never used AWS IoT SiteWise service). Customers with an existing AWS IoT SiteWise setup can enable the feature with one simple step. In the AWS IoT SiteWise console under Settings, choose Data Ingestion, and then enable Disassociated data ingestion.

Enabling the new data ingestion mode in AWS IoT SiteWise introduces a new resource called Data stream that is time series data. Data stream and asset are now resources in AWS IoT SiteWise that you can manage independently. Customers cannot disable the feature since access control to data streams and assets are managed independently. If you don’t see “Data Ingestion” setting, your account has this feature enabled already.

You can still disable the disassociated data ingestion from the gateway using access control (IAM), by writing a policy using the isAssociatedWithAssetProperty condition key. The value presented by this condition will be true if the PropertyAlias is associated with an AssetProperty, or otherwise false. This also enables a scenario where one gateway is the “development” gateway and allows all data streams, while another is the “production” gateways and does not allow disassociated data ingestion. This condition is only emitted for actions that get or put property values, such as BatchPutAssetPropertyValue.

The following example IAM policy would disable data ingestion of data streams not associated with any asset. This IAM policy consists of two statements. The first statement allows data streams that are associated to an asset property to be sent to AWS IoT SiteWise, while the second statement prevents data streams that are not associated to an asset property from being sent to AWS IoT SiteWise.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPutAssetPropertyValuesWithModeledPropertyAlias",
      "Effect": "Allow",
      "Action": "iotsitewise:BatchPutAssetPropertyValue",
      "Resource": "arn:aws:iotsitewise:*:*:time-series/*",
      "Condition": {
        "StringLikeIfExists": {
          "iotsitewise:isAssociatedWithAssetProperty": "true"
        }
      }
    },
    {
      "Sid": "DenyPutAssetPropertyValuesWithUnmodeledPropertyAlias",
      "Effect": "Deny",
      "Action": "iotsitewise:BatchPutAssetPropertyValue",
      "Resource": "arn:aws:iotsitewise:*:*:time-series/*",
      "Condition": {
        "StringLikeIfExists": {
          "iotsitewise:isAssociatedWithAssetProperty": "false"
        }
      }
    }
  ]
}

Data streams management

You can manage data streams and get instant feedback on ingested data streams through the AWS IoT SiteWise console. The data streams page has pagination and configurable page size to scale with a large number of data streams. You can filter data streams by using the prefix of the data stream name or by whether or not data streams are associated with assets. To ensure that all data streams are associated with assets, customers can use the AWS IoT SiteWise console or the ListTimeSeries API to filter for disassociated data streams. Then, you can associate those data streams with assets.

The following steps show how you can associate and disassociate data streams with asset properties.

Go to the Data streams page and select the data streams you want to associate or disassociate with asset properties as shown below.
Click “Manage data streams” button on top right, and it will take you to “Manage data streams” page.
On “Manage data streams” page as shown in the following image, you will be able to add or remove asset properties for data streams.
Clicking the “Update” button will update all changes you made, as shown in Status column.

Enhanced access control

When you read or write time-series data in AWS IoT SiteWise, that data is part of a Data stream. AWS IoT SiteWise maintains associations between AssetProperty (a DataType in SiteWise that contains asset property information), PropertyAlias (the alias of the property), and Data Stream, providing a layer of indirection so that a Data Stream can be moved without having to copy data: that is called Model Mutability.

A Data Stream must be bound to either an AssetProperty, a PropertyAlias, or both. If an AssetProperty is bound to a PropertyAlias, they both are also bound to the same Data Stream. Using the DisassociateTimeSeriesFromAssetProperty API, you can unbind a Data Stream from its AssetProperty, so that it is bound only to a PropertyAlias. Then, you can bind it to another AssetProperty by calling the AssociateTimeSeriesToAssetProperty API. You can combine this with the existing ability to change a PropertyAlias using the UpdateAssetProperty API to update your data organization within AWS IoT SiteWise with complete flexibility. You can also accomplish this through console as discussed above.

The Data Stream resource type can be used with IAM to apply different policies based on how the data are identified. When identified by PropertyAlias, a time series resource must be used. A resource for a time series looks similar to:

arn:aws:iotsitewise:region:123456789012:time-series/<DATA_STREAM_ID>

When identified by AssetId and PropertyId, it is part of an Asset Resource. The asset resource ARN (Amazon Resource Name) looks similar to:

arn:aws:iotsitewise:region:123456789012:asset/<ASSET_ID>

You can configure access control using a Data stream ARN and the PropertyAliasPrefix condition so that your Gateway can only write to Data Streams matching a given prefix.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PutAssetPropertyValuesAliasesSiteAPrefixOnly",
      "Effect": "Allow",
      "Action": "iotsitewise:BatchPutAssetPropertyValue",
      "Resource": "arn:aws:iotsitewise:*:*:time-series/*",
      "Condition": {
        "StringLikeIfExists": {
          "iotsitewise:propertyAlias": "/site-a/*"
        }
      }
    }
  ]
}

Combine this with Gateway support for automatic prefixing and it becomes easy to isolate factory sites from each other within the same account. Add in the AssetHierarchyPath condition key, and it becomes easy to carve out roles with minimal rights for both administrators and operators.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PutAssetPropertyValuesAssetsSiteBHierarchyOnly",
      "Effect": "Allow",
      "Action": "iotsitewise:BatchPutAssetPropertyValue",
      "Resource": "arn:aws:iotsitewise:*:*:asset/*",
      "Condition": {
        "StringLikeIfExists": {
          "iotsitewise:assetHierarchyPath": "/a1b2c3d4-5678-90ab-cdef-22222EXAMPLE/a1b2c3d4-5678-90ab-cdef-66666EXAMPLE"
        }
      }
    }
  ]
}

The AssetHierarchyPath is the asset’s hierarchy path. It is a string of asset IDs each separated by a forward slash. This condition key value looks similar to:

/a1b2c3d4-5678-90ab-cdef-22222EXAMPLE/a1b2c3d4-5678-90ab-cdef-66666EXAMPLE

With Disassociated Data Ingestion enabled, the AssetHierarchyPath condition is only emitted if the resource is identified in the request as an Asset; it is not emitted when identified as a Data Stream. The propertyAlias condition will be emitted if the resource is identified in the request as a Data Stream; it is not emitted when identified as an Asset.

You can look up the Data stream identifier by using the DescribeTimeSeries API or by reading metadata exported to Amazon S3. This identifier is created when data is first ingested for the Data Stream, so a caller must be authorized to the Resource arn:aws:iotsitewise:region:123456789012:time-series/* in order to create new Data streams.

Conclusion

In this blog post, we outlined recently launched features to ingest disassociated data streams and speed up time from ingestion to cloud. With industrial equipment data in the cloud, you have flexibility to associate, disassociate, and re-associate data streams with your industrial assets. Now, you can get started faster to ingest and consume your industrial data with AWS IoT SiteWise. To learn more, please visit Managing data streams and How AWS IoT SiteWise works with IAM in our User Guide.

About the Authors

Julie Zhao

Julie is a Senior Product Manager on AWS Industrial IoT team. She joined AWS in 2021 and brings three years of startup experience leading products in Industrial IoT. Prior to startups, she spent over 10 years in networking with Cisco and Juniper across engineering and product. She is passionate about building products in Industrial IoT.

Philipp Sacha

Philipp is a Specialist Solutions Architect for IoT at Amazon Web Services supporting customers in the IoT area. He joined AWS in 2015 as a general Solutions Architect and moved in 2018 into the role of a Specialist in the IoT area.

John Byrne

John is a Senior SDE with AWS SiteWise and previously with Amazon SageMaker Ground Truth. His first job, while still in high school, was at a local dial-up ISP. John has focused his 9 year career at Amazon on building highly available, highly scalable backend systems.

The Internet of Things on AWS – Official Blog