Building a foundation for GxP regulated IoT workloads on AWS

Overview

Industry 4.0 brought technology advancement in interconnectivity, automation, and analytics capabilities. This advancement coupled with Internet of Things (IoT) drives increased integration of instrument data in the cloud. As you start integrating IoT into the cloud infrastructure, there may be questions about IoT best practices and the GxP qualification process.

The objective of this blog post is to discuss best practices for GxP Qualification of IoT on Amazon Web Services (AWS). We will also be discussing about controls that can be implemented to ensure:

Security
Data integrity
Data provenance
Device management
Operational management
GxP Installation Qualification Compliance

AWS IoT Service Qualification and Reference Architecture

GxP Qualification is a risk-based approach to verify that the infrastructure is fit for intended use while ensuring that configuration changes are done without compromising the integrity of the qualification state. The scope of this blog focuses on AWS IoT service qualification. IoT devices are subject to Equipment Qualification as per FDA 21 CFR 211 and will not be discussed in this blog.

AWS provides a wide range of services to support end-to-end IoT integration. The AWS services that are used need to be qualified to demonstrate system controls. Let’s walk through an IoT reference architecture and discuss about controls that could be defined in each step.

Figure 1: An end-to-end IoT reference architecture

Step 1: Device data transmission to AWS IoT Greengrass

Capturing data is the first step in the IoT architecture. There can be various purpose-built devices which collect real-time data. Data captured from sensors or Open Platform Communications Unified Architecture (OPC UA) sources are then transmitted to AWS IoT Greengrass. Under GMP (Good Manufacturing Practice), regulatory authorities require that equipment used for the manufacturing of pharmaceutical drugs needs to be qualified before being released for their intended use.

Step 2: AWS IoT Greengrass for IoT data collection and publishing to AWS Cloud

AWS IoT Greengrass is an open source IoT edge runtime and cloud service that helps you build, deploy and manage IoT applications on your devices. Before devices can send data to AWS IoT Greengrass, you will need to create a configuration to establish connectivity. An AWS IoT Greengrass core acts as a hub, or gateway, in edge environments and manages local processes for AWS IoT Greengrass groups such as communication, shadow sync, and token exchange. This step provides connectivity between devices and AWS cloud services.

Below are some recommendations for continuous compliance within AWS IoT Greengrass service:

Encryption: Ensure encrypted traffic in transit from AWS IoT Greengrass to AWS cloud by using secured protocol set to HTTPS on port 443.
Compatibility: AWS IoT Greengrass is a component-based architecture. You will need to ensure that the AWS IoT Greengrass version is compatible between IoT devices and the core device. For a feature compatibility matrix check out Greengrass feature compatibility by operating system.
Network Firewall: A central component in AWS IoT Greengrass is called nucleus. This is where configurations are defined depending on your organization’s firewall settings. If AWS IoT Greengrass is running on a controlled environment with only outbound connectivity to limited ports, you may need to modify the default network proxy and MQTT traffic from port 8883 to port 443. This is common in most of the industrial setups.

Step 3: AWS IoT SiteWise to Ingest, Structure, Analyze and Monitor Equipment Data

AWS IoT SiteWise is a managed service that makes it easy to collect, store, organize and monitor data from industrial equipment at scale. It generates near real-time KPIs and metrics to help make better data-driven decisions. Besides the ability to compute near real-time metrics, AWS IoT SiteWise enables you to collect and monitor data across devices, identify issues with remote monitoring more quickly, and improve multi-site processes with centralized data. You can use AWS IoT SiteWise to model your physical assets and analyze the near real-time data for quicker decision making.

Figure 2: High level capabilities of AWS IoT SiteWise

AWS IoT SiteWise supports data ingestion per a number of available options. Below are some recommendations that will help you ensure security, traceability, continuous monitoring and data retention within AWS IoT SiteWise service:

Encryption: Encrypt data in transit by using Transport Layer Security (TLS) using signed certificate.
Encryption: Encrypt data at rest either by using an AWS Key Management Service (AWS KMS) key or you could choose to use a Customer Managed Key.
Data integrity: Enable logging for traceability and forensic activities (when needed).
Monitoring: Set alarms to monitor your equipment to detect when they perform outside their operating ranges.
Data Retention: Retain data per defined retention periods by enabling cold tier storage to store historical data at lower cost.

Figure 3: An example of an alarm threshold definition for continuous monitoring

Step 4: Securely Connects Devices at Scale using AWS IoT Core

Data from a MQTT bridge component will relay messages between local devices and AWS IoT Core. AWS IoT Core lets you connect your IoT devices and route messages to AWS cloud services.

These are a few recommendations that will help you ensure security, traceability, and continuous monitoring within AWS IoT Core:

Encryption: Encrypt data in transit by using TLS using X.509 signed certificate attached to devices. Certificates provide AWS IoT Core with the ability to authenticate the client and device connections.
Security: Apply least privilege permissions to IoT policies. The permissions that you define in IoT policies determines permissions for authentication, connectivity and message publishing between devices and AWS IoT Core.
Data Integrity: Enable logging for traceability and forensic activities (when needed).
Monitoring: Enable monitoring with a pre-created Amazon CloudWatch dashboard displaying metrics from registered devices. You can build additional metrics and custom dashboards as needed.

Figure 4: Dashboard displays the number of messages published and received by devices aggregated by protocol, type, and messages published over time

Step 5: Security and Compliance in AWS cloud

Security is foundational to every IoT application. You should design your IoT security implementation per the AWS Well Architected Framework with IoT Lens. Besides security in the AWS cloud, it is important to ensure that devices implement hardware security best practices such as performing regular patches to remove vulnerabilities.

Below are additional security design recommendations:

Use AWS IoT Device Defender to audit IoT configurations and detect device anomalies within defined ranges.
Use AWS Secrets Manager for deploying secrets to AWS IoT Greengrass core devices.
Encrypt Data-at-Rest and in-Transit.
Enforce certificate-based authentication.

AWS IoT services provide a set of Identity and Access Management (IAM) policies to use as-is or you can create custom IAM policies. These policies allow access to configuration and data operations. Access and permission should be set according to the Least Privilege Principle. The policies provided should be the minimal set of permissions required to perform the task. IAM roles should be used to delegate permissions.

Step 6: Data Storage

IoT workloads are often designed to generate large quantities of data. Data from AWS IoT SiteWise and AWS IoT Core is stored in Amazon Simple Storage Service (Amazon S3) for a variety of purposes such as:

Analytics
Machine learning
Regulatory purposes

You can harness the full value of the data once it reaches Amazon S3. Let’s walk through some storage service compliance recommendations for both AWS IoT SiteWise and AWS IoT Core.

AWS IoT SiteWise Data Storage

AWS IoT SiteWise supports two storage tiers for equipment data: a hot tier optimized for real-time applications, and a cold tier optimized for analytical applications. The hot tier stores frequently accessed data with lower latency for faster access to the real-time measurement values from your equipment. The cold tier stores less-frequently accessed data that can tolerate higher read latency for lower storage cost. AWS IoT SiteWise data is stored in a hot tier by default. In order to enable cold tier storage, you will need to set a data retention period in AWS IoT SiteWise. By enabling cold tier storage in AWS IoT SiteWise, you will be able to lower storage cost on infrequently accessed historical data while retaining a copy of the data for:

backup,
historical reporting and
advanced analytics purposes such as artificial intelligence (AI) and machine learning (ML) training.

Once cold tier storage is enabled, AWS IoT SiteWise will export data from measurements, metrics, transforms, and aggregates to the customer managed Amazon S3 bucket every 6 hours. In addition, AWS IoT SiteWise will export to your Amazon S3 bucket any changes to asset and asset model definitions within minutes. You will need to ensure Data Retention is defined and the access role has least privileged policy.

AWS IoT Core Data Storage

You can configure an AWS IoT rule action to store MQTT messages into an Amazon S3 bucket. Parameters that you will need to provide while setting up the rule action to store messages into an Amazon S3 are:

an Amazon S3 Bucket,
an Amazon S3 path to the file where the data is written,
an IAM role that allows access to the Amazon S3 bucket, and
a SQL statement to select attributes and apply topic filter as shown in the SQL statement below.

SELECT <Attribute> FROM <Topic Filter> WHERE <Condition>

While Amazon S3 is one of the possible destinations for low frequency data, you can also invoke the provided AWS IoT rule actions to send data to other AWS services.

Amazon Simple Storage Service (Amazon S3)

Since Amazon S3 is the centralized location that stores data from AWS IoT SiteWise and AWS IoT Core, it is crucial to apply controls and best practices to the defined bucket. Below are some security recommendations for the Amazon S3 bucket:

Data Protection: Ensure the bucket is not publicly accessible.
Data Protection: Enable bucket versioning.
Data Protection: Enable bucket server-side encryption.
Data Integrity: Enable bucket logging for storing server access logs.

Step 7: Automated Infrastructure IQ Report Generation (Optional)

One of the benefits of cloud computing is the ability to script and automate resource deployment through Infrastructure as Code tools such as AWS CloudFormation or Terraform. The granular details captured in the deployment process, along with the ability to retrieve metadata of resources, enables AWS IoT Installation Qualification automation.

The Installation Qualification (IQ) process can be automated to retrieve metadata of deployed resources within the qualification scope by using common tag values. It compares the actual configuration values with expected values. This process iterates through all the deployed resources and generates a PDF report containing details of the deployed resources and verification results. It compares attributes of deployed resources against planned attributes documented in the Configuration Specifications. The PDF report generation process can be setup to run as needed or upon any resource changes. Having an automated solution for infrastructure IQ improves consistency due to automation of development, deployment and testing processes.

Besides the benefits stated above, the following are some of the advantages of automating AWS IoT infrastructure IQ process:

Cost reduction: Cost of managing infrastructure compliance can be reduced by decreasing manual effort of updating documentation.
Version control: The infrastructure environment can be governed by having the Infrastructure as Code script version controlled and roll back to the last good state upon failures.
Repeatable: Infrastructure that you can replicate, re-deploy, and re-purpose. This solution can be centralized at a master account, organization unit or per app basis.

Figure 5: Example listing of deployed IoT resources in a sample automated Installation Qualification report

Figure 6: Example listing of build specification verifications in a sample automated Installation Qualification report

Conclusion

In this blog, we walked through an IoT reference architecture and discussed about controls that can be defined. The steps provided above covers end-to-end data flow of IoT in AWS Cloud at a high level. AWS IoT services provide various capabilities to enforce compliance and maintain the GxP qualification state over deployed resources. By following the reference architecture as discussed above, it will enable you to collect, store, and analyze your IoT data securely at scale. You could then unlock the data potential in AWS Cloud by integrating with other AWS services for an end-to-end solution. Also, the automated Installation Qualification process can help reduce qualification cycle time, manual effort, and improve infrastructure qualification accuracy.

As you begin your journey in designing IoT architecture for regulated workloads, we strongly suggest that you review the AWS Well Architected Framework with IoT Lens for IoT best practices and to better understand business impacts with every design decision. It contains broader best practices and controls for IoT implementation. The AWS Well Architected Framework is based on five pillars which are operational excellence, security, reliability, performance efficiency, and cost optimization. For help in instituting the reference architecture, please contact your AWS Account team.

Security and compliance are a shared responsibility between AWS and each customer. For further reading, please refer to Securing Internet of Things (IoT) with AWS and the whitepapers on GxP Systems on AWS. For in-depth industrial IoT architectural patterns, please refer to Industrial IoT Architecture Patterns.