Collecting data from industrial devices to AWS Services

Industrial data, a valuable resource, offers insights into machine performance, processes, and products. By collecting and analyzing this data, companies can enhance product quality, optimize efficiency, manage supply chains, and effectively schedule preventive maintenance.

However, collecting data from industrial devices can be a complex process. Typically, these devices are manufactured by different vendors, each using their own proprietary data formats, addressing schemes, and access protocols. Also, networking restrictions and platform requirements can further complicate the operation and deployment of the required software in production environments.

Shop Floor Connectivity (SFC) is a data ingestion enabler, that can quickly deliver customizable greenfield & brownfield connectivity solutions. Addressing the limitations of and unifying data collection from existing IoT data collection services, SFC allows customers to collect data from equipment from different vendors in a consistent manner, for use with various AWS services.

SFC Components

There are three main types of components that make up SFC.

Protocol Adapters
SFC Core
Target Adapters

Overview of the SFC Components

Protocol Adapters

Protocol adapters are used to read data from one or more industrial devices. This adapter interface abstracts the used protocol from the devices and delivers the data with additional metadata in a common format to the SFC Core. The interface is designed so that SFC can be easily extended with new protocol adapters without any modifications to the rest of the framework.

At the time of this article, SFC has adapters for Modbus TCP, MQTT, OPCUA, Ethernet/IP PCCC, ADS, SNMP, S7, and SQL.

Core

The SFC Core component is the controller of the SFC Framework. It handles configuration and scheduling of the data collection through the protocol adapters. It can optionally transform each received data value using a combination of one or more of the over 60 available transformation functions, which can address complex data transformation requirements. The core has end-to-end data type fidelity, meaning the data can be sent to the targets in the data format it was read from the source, including complex structured datatypes and multidimensional arrays.

Optionally, the data can be buffered and aggregated at the edge to reduce network traffic by using one or more of the 12 available aggregation functions.

The core integrates with AWS Secrets Manager to allow the use of placeholders for secrets used in its configuration.

Target Adapters

Target adapters are components that receive the data from the SFC Core and send it to their specific AWS or local services. Components can optionally apply data transformations using an Apache Velocity template to deliver the data in the required format for the receiving service. At the time of writing, there are adapters for the following AWS services: AWS IoT Analytics, AWS IoT Core, Amazon Kinesis Data Streams, Amazon Data Firehose, AWS Lambda, Amazon Managed Streaming for Apache Kafka (MSK), Amazon Simple Storage Service (S3), AWS IoT SiteWise, Amazon Timestream, Amazon Simple Notification Service (SNS), and Amazon Simple Queue Service (SQS). Additional targets for the local filesystem, terminal output, and MQTT clients are also available.

Data Collection

Data collection is set up through configuration and does not require additional coding. This is accomplished by defining one or more schedules that are executed by the SFC core process. Each schedule defines the interval and the sources, which can be of different types, from which data of different types must be read. A source defines the protocol adapter that is used and the values that must be read using that adapter. Optionally, transformations and aggregations can be defined and applied to the collected data. A schedule also specifies one or more target adapters to send the collected and processed data to.

Deployment

The SFC core can be deployed as a single process that loads the configured connectors and target adapters configured by the user. Alternatively, protocol adapters, target adapters and the SFC Core can be deployed as individual services that communicate via a TCP/IP stream to exchange the collected data.

These services can reside on the same physical machine or on separate external hardware. This flexibility enables load balancing for data collection and processing, and it allows deployment across network segments with varying connectivity requirements and restrictions. The diagram below shows some example deployments.

SFC deployment examples

SFC components require a Java Virtual Machine to execute. Components can be executed as a standalone application, container or as a AWS IoT Greengrass component.

Extending SFC

SFC can be extended by implementing new protocols and target adapters in a straightforward process. In order to build a component for a new adapter, developers must implement the interface for the adapter which is used by the SFC Core to communicate with the adapter. SFC is designed to allow developers to focus on protocol and target implementations through the use of SFC’s code, but without requiring modification to the SFC’s code.

SFC also allows the configuration of customized configuration providers, enabling configuration from existing configuration data stores. Metrics generated by SFC components can be sent to Amazon CloudWatch metrics or to other metric data stores using a configurable custom metrics writer.

Logging output data can also be written to a custom store by configuring a custom log writer.

Example configuration OPC UA to Amazon S3 and AWS IoT Core

This is a simple example configuration that reads data from an OPC UA server and sends it directly to a MQTT topic in the AWS IoT Core service and objects in an Amazon S3 bucket without any further processing or aggregation. Additionally, it includes a Debug target that is not shown in the image below, which sends the data to the console for debugging purposes.

SFC configuration file

{
"AWSVersion": "2022-04-02",
"Name": "OPCUA to S3, IoT Core and console",
"Schedules": [
    {
      "Name": "OPCUA-DATA",
      "Interval": 1000,
      "Sources": {
        "OPCUA-SOURCE": ["*"]
      },
      "Targets": ["IoTCoreTarget", "S3Target", "DebugTarget"]
    }
],
"Sources": {
    "OPCUA-SOURCE": {
      "Name": "OPCUA-SOURCE",
      "ProtocolAdapter": "OPC-UA",
      "AdapterOpcuaServer": "OPCUA-SERVER",
      "SourceReadingMode": "Subscription",
      "Channels": {
        "ServerStatus": {
          "NodeId": "ns=0;i=2256"
        },
        "SimulationCounter": {
          "NodeId": "ns=3;i=1001"
        },
        "SimulationRandom": {
          "NodeId": "ns=3;i=1002"
        },
        "LevelAlarm": {
          "NodeId": "ns=6;s=MyLevel.Alarm",
          "EventType": "ExclusiveLevelAlarmType"
        }
      }
    }
},
"ProtocolAdapters": {
    "OPC-UA": {
      "AdapterType": "OPCUA",
      "OpcuaServers": {
        "OPCUA-SERVER": {
          "Address": "opc.tcp://sfc-server",
          "Path": "OPCUA/SimulationServer"
        }
      }
    }
},

"Targets": {
    "IoTCoreTarget": {
      "TargetType": "AWS-IOT-HTTP",
      "TopicName": "sfc-data-topic",
      "Region": "eu-west-1",
      "CredentialProviderClient": "AwsIotClient"
    },
    "S3Target": {
      "TargetType": "AWS-S3",
      "Region": "eu-west-1",
      "BucketName": "sfc-data-bucket",
      "Interval": 60,
      "BufferSize": 1,
      "CredentialProviderClient": "AwsIotClient",
      "Compression": "Zip"
    },
    "DebugTarget": {
      "TargetType": "DEBUG-TARGET"
    }
},

"AdapterTypes": {
    "OPCUA": {
      "JarFiles": ["/sfc/opcua/lib"],
      "FactoryClassName": "com.amazonaws.sfc.opcua.OpcuaAdapter"
    }
},

"TargetTypes": {
    "AWS-IOT-HTTP": {
      "JarFiles": ["/sfc/aws-iot-http-target/lib"],
      "FactoryClassName": "com.amazonaws.sfc.awsiot.http.AwsIotHttpTargetWriter"
    },
    "AWS-S3": {
      "JarFiles": ["/sfc/aws-s3-target/lib"],
      "FactoryClassName": "com.amazonaws.sfc.awss3.AwsS3TargetWriter"
    },
    "DEBUG-TARGET": {
      "JarFiles": ["/sfc/debug-target/lib"],
      "FactoryClassName": "com.amazonaws.sfc.debugtarget.DebugTargetWriter"
    }
},

"AwsIotCredentialProviderClients": {
    "AwsIotClient": {
      "IotCredentialEndpoint": "1234567890abcd.credentials.iot.eu-west-1.amazonaws.com",
      "RoleAlias": "TokenExchangeRoleAlias",
      "ThingName": "MyThingName",
      "Certificate": "/sfc/cert/certificate.crt",
      "PrivateKey": "/sfc/cert/private.key",
      "RootCa": "/sfc/cert/root.pem"
    }
}
}

Configuration key elements

Schedule: The configuration revolves around schedules, exemplified by ‘OPCUA-DATA’ in this instance. This schedule outlines the sources for data retrieval, sets the collection interval in milliseconds, and designates the target destinations for data delivery.

Sources: This section outlines the sources for data extraction. In this illustration, a single OPC UA source is defined. Within this source, details include the adapter name and the specific server within that adapter from which the data is acquired. The channels embody concrete data items, tailored to different types of source adapters. In the context of the OPC UA adapter in this example, these channels represent data items like data nodes, events, or alarms – each identified by its unique node ID.

Protocol Adapters: The protocol adapters being used are defined in this section. Each protocol adapter type has its own specific elements. In the case of the used OPC UA adapter, there is a definition of the actual OPC UA server from which the data for a source is read.

Targets: In this section, the destinations to which data can be delivered are defined. Each type of target has its specific elements, such as the name of the Amazon S3 bucket for the Amazon S3 target or the topic name for the AWS IoT Core adapter.

Adapter Types and Target Types: Each adapter and target definition refer to their adapter or target types, which need to be configured in the AdapterTypes or TargetTypes section. Each type defines from where the JAR files that implement the adapter or the target can be loaded and the name of the factory class to create instances of these. The SFC Core will use this information to create instances of the required adapters and targets. (Note: these sections are only required when the adapter or target is running in the same process as the SFC Core. If an adapter or target is executed as an IPC service, the adapter refers to a protocol or target server, which defines the address and port where it can be reached.)

AWS IoT Credential Provider Clients: This section defines the clients used by targets that need access to AWS services. Each client uses a set of X.509 certificates and key files to obtain temporary credentials. The role alias points to a role that grants access to AWS services. See the AWS Security blog post “How to Eliminate the Need for Hardcoded AWS Credentials in Devices by Using the AWS IoT Credentials Provider” for detailed information.

Getting started with SFC

SFC is provided as source code and can be obtained from https://github.com/aws-samples/shopfloor-connectivity. The SFC repository on GitHub includes extensive documentation, including detailed examples for configuring SFC and creating custom components. Pre-built SFC packages can be downloaded from https://github.com/aws-samples/shopfloor-connectivity/releases.

The SFC repository also includes a comprehensive quickstart guide that outlines the steps required to deploy and configure SFC for the example used in this article.

AWS for Industries