[SEO Subhead]
This Guidance combines AWS IoT SiteWise with HighByte Intelligence Hub to facilitate the transformation of manufacturing and industrial operations through a proven, governed, data-driven approach. HighByte Intelligence Hub serves as an edge-native DataOps solution, bridging the gap between operational technology (OT) and IT by integrating industrial information across diverse systems. With this Guidance, you can achieve a Unified Namespace (UNS) that helps ensure seamless data access and integration throughout the manufacturing enterprise's lifecycle.
This Guidance facilitates the establishment and upkeep of an enterprise-governed asset model across remote sites with central and remote HighByte Intelligence Hubs. The central Intelligence Hub enables the creation and maintenance of an enterprise-governed asset hierarchy and model across remote sites, while remote Intelligence Hubs at the site level aid in ingesting data from different industrial edge data sources. This provides an effortless publishing process to AWS IoT SiteWise, a purpose-built AWS industrial Internet of Things (IoT) service.
Please note: [Disclaimer]
Architecture Diagram
1: Delivering Industrial DataOps (IDO) on Industrial Data Fabric (IDF)
This high-level architecture diagram is a reference that helps you create an enterprise governed model, ingest near real-time and historical data at scale from edge data sources into an IDF on AWS, and interface with applications using REST APIs.
Step 1
Define a governed data model in a central location. The centralized model can be populated with metadata from corporate systems such as enterprise resource planning (ERP), piping and instrumentation diagram (P&ID), and computerized maintenance management system (CMMS).
Step 2
Bring the governed model into an edge application, where additional metadata from systems, such as a manufacturing execution system (MES), can be added. Data streams can then be mapped to the imported model. Asset Hierarchy can also be moved to AWS IoT SiteWise.
Step 3
Ingest data depending on the source. Stream data feeds from a variety of sources using AWS DataSync for file share, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK), AWS IoT Core, AWS IoT SiteWise, or AWS Transfer Family for SFTP.
Step 4
Optimize data storage for the workload, which can include Amazon DynamoDB key value and document data structures, Amazon Simple Storage Service (Amazon S3) for object storage, Amazon Neptune for graph use cases, Amazon Redshift for data warehousing, Amazon Timestream for time series data, and AWS IoT SiteWise to organize industrial equipment data. Data from AWS IoT SiteWise can be integrated with AWS IoT TwinMaker.
Step 5
Use AWS artificial intelligence and machine learning (AI/ML) services such as Amazon SageMaker to build, train, and deploy ML models.
Step 6
Use AWS analytics services such as Amazon OpenSearch Service, Amazon Managed Service for Apache Flink, Amazon Athena, Amazon EMR, and AWS Glue for data processing.
Step 7
AWS WAF provides protection from web exploits while Amazon API Gateway provides REST method support. Amazon Cognito provides user authentication and AWS token support.
Step 8
Egress data from the Industrial Data Fabric (IDF) with connectors directly to AWS services or through API Gateway to supported AWS services. For services that are not supported by API Gateway, an OAuth 2.0 pattern with Amazon Cognito is used to generate AWS temporary tokens.
Step 1
Define a governed data model in a central location. The centralized model can be populated with metadata from corporate systems such as enterprise resource planning (ERP), piping and instrumentation diagram (P&ID), and computerized maintenance management system (CMMS).
Step 2
Bring the governed model into an edge application, where additional metadata from systems, such as a manufacturing execution system (MES), can be added. Data streams can then be mapped to the imported model. Asset Hierarchy can also be moved to AWS IoT SiteWise.
Step 3
Ingest data depending on the source. Stream data feeds from a variety of sources using AWS DataSync for file share, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK), AWS IoT Core, AWS IoT SiteWise, or AWS Transfer Family for SFTP.
Step 4
Optimize data storage for the workload, which can include Amazon DynamoDB key value and document data structures, Amazon Simple Storage Service (Amazon S3) for object storage, Amazon Neptune for graph use cases, Amazon Redshift for data warehousing, Amazon Timestream for time series data, and AWS IoT SiteWise to organize industrial equipment data. Data from AWS IoT SiteWise can be integrated with AWS IoT TwinMaker.
Step 5
Use AWS artificial intelligence and machine learning (AI/ML) services such as Amazon SageMaker to build, train, and deploy ML models.
Step 6
Use AWS analytics services such as Amazon OpenSearch Service, Amazon Managed Service for Apache Flink, Amazon Athena, Amazon EMR, and AWS Glue for data processing.
Step 7
AWS WAF provides protection from web exploits while Amazon API Gateway provides REST method support. Amazon Cognito provides user authentication and AWS token support.
Step 8
Egress data from the Industrial Data Fabric (IDF) with connectors directly to AWS services or through API Gateway to supported AWS services. For services that are not supported by API Gateway, an OAuth 2.0 pattern with Amazon Cognito is used to generate AWS temporary tokens.
-
2: IDF Governed Data Model with HighByte Intelligence Hub
This architecture diagram helps you create an enterprise governed data model, ingest real-time and historical data at scale from edge data sources, and visualize this data using Amazon Managed Grafana. AWS IoT SiteWise provides a contextualized timeseries data store and AWS IoT TwinMaker allows for digital twin scenes.
Click to enlarge
Step 1
Define a governed data model in HighByte Central Hub running as a Docker image in Amazon Elastic Container Service (Amazon ECS). Publish the Enterprise Hierarchy from HighByte to AWS IoT SiteWise, including model information. Sync asset models from the central hub to the remote hubs.Step 2
Connect to data sources with the HighByte remote hubs using models from the central hub. Insert data into AWS IoT SiteWise in the appropriate portion of the hierarchy model. Data can be inserted in near real-time using AWS IoT SiteWise or delayed using buffered ingestion.Step 3
Use buffered ingestion through Amazon S3 to reduce AWS IoT SiteWise ingestion costs. The AWS IoT SiteWise Open Database Connectivity (ODBC) driver provides direct integration to AWS IoT SiteWise for business intelligence (BI) client tools.Step 4
Train models in Amazon Lookout for Equipment for anomaly detection and prediction maintenance using data from AWS IoT SiteWise. Native AWS IoT SiteWise integration through Amazon S3 allows data exchange between services. Use AWS AI/ML services such as SageMaker to build, train, and deploy custom ML models.Step 5
Access AWS IoT SiteWise data through API Gateway and AWS Lambda with Amazon Cognito for authenticated REST method support.Step 6
Visualize AWS IoT SiteWise data and AWS IoT TwinMaker scenes using Amazon Managed Grafana.Step 1
Define a governed data model in HighByte Central Hub running as a Docker image in Amazon Elastic Container Service (Amazon ECS). Publish the Enterprise Hierarchy from HighByte to AWS IoT SiteWise, including model information. Sync asset models from the central hub to the remote hubs.Step 2
Connect to data sources with the HighByte remote hubs using models from the central hub. Insert data into AWS IoT SiteWise in the appropriate portion of the hierarchy model. Data can be inserted in near real-time using AWS IoT SiteWise or delayed using buffered ingestion.Step 3
Use buffered ingestion through Amazon S3 to reduce AWS IoT SiteWise ingestion costs. The AWS IoT SiteWise Open Database Connectivity (ODBC) driver provides direct integration to AWS IoT SiteWise for business intelligence (BI) client tools.Step 4
Train models in Amazon Lookout for Equipment for anomaly detection and prediction maintenance using data from AWS IoT SiteWise. Native AWS IoT SiteWise integration through Amazon S3 allows data exchange between services. Use AWS AI/ML services such as SageMaker to build, train, and deploy custom ML models.Step 5
Access AWS IoT SiteWise data through API Gateway and AWS Lambda with Amazon Cognito for authenticated REST method support.Step 6
Visualize AWS IoT SiteWise data and AWS IoT TwinMaker scenes using Amazon Managed Grafana. -
3: HighByte Intelligence Hub Industrial DataOps on AWS
This architecture diagram demonstrates how HighByte Intelligence Hub integrates OT with IT to combine industrial information across multiple systems, enabling OT teams to model, transform, and share plant floor data with IT systems.
Click to enlarge
Step 1
HighByte Intelligence Hub consumes both real-time and asset model data from a myriad of edge data sources, including relational databases and AWS IoT Greengrass, by using standard industrial protocol input connectors. This includes data ingestion from industrial historians, such as Inductive Automation’s Ignition Server and Aveva’s PI System.Step 2
Intelligence Hub enables customers to standardize, organize, and merge your industrial data into a single equipment model. Then, using flows, you can route the asset models to multiple output connectors, each with a different frequency.HighByte can be deployed on the edge using an AWS container option, including IoT Greengrass, Amazon Elastic Kubernetes Service (Amazon EKS) Anywhere, and Amazon ECS Anywhere. Intelligence Hub provides a variety of output connectors that support many of standard industrial protocols, including a native AWS IoT SiteWise connector.
Step 3
Bring in asset models and timeseries based sensor data into Amazon S3. AWS Lake Formation helps you collect and catalog data from databases and object storage, move the data into Amazon S3, and clean and classify data using ML algorithms. Data is accessed through AWS Glue Data Catalog.Step 4
Build asset models within the HighByte editor and deploy the model directly to AWS IoT SiteWise along with the streaming data. Calculate and visualize metrics from telemetry data using AWS IoT SiteWise Monitor.Step 5
Intelligence Hub can connect directly to AWS IoT Core through its native Message Queuing Telemetry Transport (MQTT) service or use IoT Greengrass locally. Intelligence Hub also enables bi-directional communication with AWS IoT Core and IoT Greengrass.Step 6
Intelligence Hub can connect directly to Amazon Kinesis Data Streams for massively scalable and durable near real-time data streaming.Streaming data can be transformed and analyzed in near real-time using Amazon Managed Service for Apache Flink, and sent to Amazon Data Firehose. Time series data can also be sent to Timestream from Amazon Managed Service for Apache Flink.
Step 7
Telemetry data is published in near real-time to Data Firehose by either an AWS IoT Core rule, Kinesis Data Streams, or a HighByte and Data Firehose connector. This loads the streaming data reliably into an Amazon S3 data lake.Step 8
Use Amazon Redshift to store structured data sets and analytics results in a data warehouse. Data into Amazon Redshift can be ingested either through AWS Glue from Amazon S3 or directly through the Intelligence Hub Redshift connector.Step 9
Create BI reports and visualize data from Amazon Redshift and Amazon S3 with Amazon QuickSight and Athena.Step 10
When real-time and historical data is available in Amazon S3, Lookout for Equipment uses the data to detect abnormal equipment behavior so that potential machine failures are detected before failures occur and unplanned downtime is avoided.Computed metrics can be written back into Amazon S3 for storage and consumption. Custom ML models can be developed with SageMaker.
Step 1
HighByte Intelligence Hub consumes both real-time and asset model data from a myriad of edge data sources, including relational databases and AWS IoT Greengrass, by using standard industrial protocol input connectors. This includes data ingestion from industrial historians, such as Inductive Automation’s Ignition Server and Aveva’s PI System.Step 2
Intelligence Hub enables customers to standardize, organize, and merge your industrial data into a single equipment model. Then, using flows, you can route the asset models to multiple output connectors, each with a different frequency.HighByte can be deployed on the edge using an AWS container option, including IoT Greengrass, Amazon Elastic Kubernetes Service (Amazon EKS) Anywhere, and Amazon ECS Anywhere. Intelligence Hub provides a variety of output connectors that support many of standard industrial protocols, including a native AWS IoT SiteWise connector.
Step 3
Bring in asset models and timeseries based sensor data into Amazon S3. AWS Lake Formation helps you collect and catalog data from databases and object storage, move the data into Amazon S3, and clean and classify data using ML algorithms. Data is accessed through AWS Glue Data Catalog.Step 4
Build asset models within the HighByte editor and deploy the model directly to AWS IoT SiteWise along with the streaming data. Calculate and visualize metrics from telemetry data using AWS IoT SiteWise Monitor.Step 5
Intelligence Hub can connect directly to AWS IoT Core through its native Message Queuing Telemetry Transport (MQTT) service or use IoT Greengrass locally. Intelligence Hub also enables bi-directional communication with AWS IoT Core and IoT Greengrass.Step 6
Intelligence Hub can connect directly to Amazon Kinesis Data Streams for massively scalable and durable near real-time data streaming.Streaming data can be transformed and analyzed in near real-time using Amazon Managed Service for Apache Flink, and sent to Amazon Data Firehose. Time series data can also be sent to Timestream from Amazon Managed Service for Apache Flink.
Step 7
Telemetry data is published in near real-time to Data Firehose by either an AWS IoT Core rule, Kinesis Data Streams, or a HighByte and Data Firehose connector. This loads the streaming data reliably into an Amazon S3 data lake.Step 8
Use Amazon Redshift to store structured data sets and analytics results in a data warehouse. Data into Amazon Redshift can be ingested either through AWS Glue from Amazon S3 or directly through the Intelligence Hub Redshift connector.Step 9
Create BI reports and visualize data from Amazon Redshift and Amazon S3 with Amazon QuickSight and Athena.Step 10
When real-time and historical data is available in Amazon S3, Lookout for Equipment uses the data to detect abnormal equipment behavior so that potential machine failures are detected before failures occur and unplanned downtime is avoided.Computed metrics can be written back into Amazon S3 for storage and consumption. Custom ML models can be developed with SageMaker.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaperThe majority of AWS services used by this Guidance, such as Amazon S3 and API Gateway, are serverless, lowering the operational overhead of maintaining the Guidance. This also allows you to evolve the design pattern in a continuous cycle of improvement over time.
-
SecurityRead the Security whitepaper
This Guidance leverages AWS Security Token Service (AWS STS) and Amazon Cognito. These services allow you to take advantage of cloud technologies to protect data, systems, and assets in a way that can improve your security posture.
Security Best Practices for Manufacturing OT describes how to design, deploy, and secure distributed manufacturing workloads and resources at the industrial edge.
-
ReliabilityRead the Reliability whitepaper
This Guidance uses many of the AWS managed services to allow for a highly available network topology. Availability and reliability are managed on your behalf by AWS service teams (for example, Amazon S3, AWS IoT SiteWise, and Amazon Cognito).
-
Performance EfficiencyRead the Performance Efficiency whitepaper
This Guidance uses purpose-built storage services, such as Amazon S3, that can reduce latency and increase throughput. You can use cross-region replication (CRR) to provide lower-latency data access to different geographic Regions. This Guidance provides multiple data-driven approaches to meet your workload requirements of scaling, traffic, and data access patterns.
-
Cost OptimizationRead the Cost Optimization whitepaper
This Guidance uses purpose-built storage services, such as Amazon S3, that can reduce latency and increase throughput.
-
SustainabilityRead the Sustainability whitepaper
This Guidance utilizes scalable services, such as Amazon S3, to align the services to your needs. Its functionalities are implemented by using a serverless architecture (including Amazon Cognito and API Gateway). Your resources are available only when needed and do not run constantly.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
HighByte Intelligence Hub
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.