Post by David McClellan, P.E., Engineering Manager and Jason Crozier, Cloud Architect at Terex.

Genie®, a Terex® Brand, is a global manufacturer of lifting and material handling products and services delivering lifecycle solutions that maximize customer return on investment.

In this post, we discuss how Genie used an AWS IoT SiteWise based solution to ingest, organize, and analyze critical process parameters from the paint system. This solution identified inconsistent and improper pretreatment parameters in near real-time and enabled Genie to apply necessary corrective actions to improve the downstream paint quality of Genie lifts.

AWS IoT SiteWise is a managed service to collect, organize, and analyze data from sensors, equipment, machines, programmable logic controllers (PLC) on the plant floor at scale.

About the Use Case

Genie had a high priority need to collect data from the paint pretreatment process in near real-time. This was due to increased paint-related defects as reported by Genie Customers. The data revealed that the paint pretreatment process was one key contributor to the paint-related defects.

Before using AWS, Genie used manual methods to collect shop floor data stored on hand written logs or electronic spreadsheets. Process instrumentation was in place but the readable outputs from this instrumentation were not being used. Genie was moving toward a traditional database-based solution to house the process instrumentation data and developing in-house websites to visualize the data.

The following principles were critical for the solution:

• Data must be stored for up to 10-years for historical trend analysis
• Genie team members must be able to dive deep into the historical data for specific dates of manufacturing
• Data must be visualized in near real-time on a dashboard, so machine operators and Engineering can react quickly to irregular trends or out of bound conditions for the key process parameters and key performance indicators (KPI)
• Paint operators must be able to manually enter collected data from the shop floor such as results from a chemical testing process (titration)
• The solution must be able to generate alerts and send notifications in near real-time for out of bound operating conditions to take necessary corrective actions
• Solution must be scalable and repeatable for rapid deployment globally at scale

After collaborating with AWS Professional Services for the pilot solution, we also quickly realized the potential of unlimited analytics opportunities for data-driven insights. We could do this by connecting related enterprise data from systems such as Enterprise Resource Planning (ERP) and Manufacturing Execution System (MES), with the operational data from manufacturing.

Solution Walkthrough

Genie and AWS Professional Services deployed an end-to-end production ready solution in eight weeks for the manufacturing operations, delivering real business outcomes. The following are core functions delivered by the solution:

• Securely connect the plant floor to AWS
• Ingest paint pre-treatment process data to AWS in near real-time
• Ingest manually collected data (such as titration results) from the shop floor to AWS
• Create virtual assets of the sensors and key processes from the shop floor in the AWS Cloud
• Visualize KPIs for the assets in near real-time to identify inconsistent and improper paint pretreatment
• Monitor operating conditions in near real-time, create alerts, and send notifications for corrective actions
• Enable Business Intelligence (BI) reporting for trend analysis and dive deep on historical data of up to 10-years

The following diagram illustrates the end-to-end solution deployed along with the AWS services used:

This solution is foundational for Genie’s digital innovation journey. The following diagram illustrates how the solution can be extended to other enterprise data (ERP, MES, secondary sensors, other IoT devices) to perform advanced analytics such as Machine Learning and Data Warehousing on product, enterprise, and manufacturing data for optimizing operations.

This solution walkthrough consists of the following considerations:

1. Plant to AWS connectivity
2. Security first
3. Ingesting data into AWS
4. Creating asset-hierarchy in the cloud
5. Storing data in the cloud
6. Creating alerts and send notifications for out of bound conditions
7. Visualizing KPIs in near real-time and performing historical analysis
8. Repeatable for rapid deployment at scale globally

Plant to AWS connectivity

The management of the edge gateway remotely was a key requirement for the AWS IoT Greengrass service and KEPServerEX software running on the gateway. Not all industrial edge gateways provide a remote access controller, thereby requiring manual in-person intervention for a power recycle. To satisfy the requirement, an on-premise server was chosen with the following specifications:

• Form Factor 1U
• Short Chassis depth to place into a network Rack
• Rugged server with dust filters for survival in the manufacturing shop floor
• Remote access controller allowing remote management regardless of the OS state
• Ability to deploy AWS IoT Greengrass for multiple manufacturing sites via existing setup with a golden image
• Setup for automatic failover

For high availability (HA), the on-premise server was joined to the existing remote access controller that enabled faster failover using regularly taken backups, reducing downtime. The backup can be restored when needed to the existing on-premises servers and retain the same IP address avoiding configuration updates for AWS IoT Greengrass. The automatic failover can be configured on the existing setup.

While the entire process instrumentation connectivity from the shop floor to KEPServerEX was established in a few hours, live data streaming into AWS IoT SiteWise in the cloud happened within a week.

Security first

Security is job zero at Genie. A secure network topology and authentication system were implemented. AWS IoT Greengrass uses X.509 certificates, AWS IoT policies, and IAM policies and roles to secure the applications that run on the edge gateway locally. Refer to Security in AWS IoT Greengrass to learn more on how security on AWS is a shared responsibility.

Ingesting data into AWS

Paint pretreatment process data from all the stages are ingested into AWS IoT SiteWise over the OPC-UA protocol. This uses the AWS IoT SiteWise Connector software running on the edge gateway, which is also running AWS IoT Greengrass.

In addition to the paint pretreatment process data, paint operators are able to enter manually collected data from process and instrumentation checks. This includes results from the chemical testing process (titration) that were also ingested into AWS IoT SiteWise using its BatchPutAssetPropertyValue API in an event-driven way.

Creating asset-hierarchy in the cloud

Using AWS IoT SiteWise asset modeling capability, Genie’s goal was to set up the following conceptual asset hierarchy at the enterprise level. This enabled a cross-site view of the key operational KPIs and contextualized the telemetry data for downstream applications.

The following asset hierarchy is set up in AWS IoT SiteWise for the pilot plant:

Storing data in the cloud

The modeled data from paint pretreatment process is stored in a scalable and managed time-optimized data store of AWS IoT SiteWise (often referred to as a “hot” operational data store). As the data becomes warm or cold, AWS IoT SiteWise is configured to send the data to Amazon S3 (data lake) to build the 10-years of history for downstream analytics.

Once the data is in the data lake, options are:

• Analyze the data using familiar SQL with Amazon Athena
• Perform Machine Learning on the data using Amazon SageMaker
• Perform data integration, enrichments, and data transformations between operational and enterprise data (eg. MES, ERP) available in the data lake using Amazon EMR, AWS Glue etc.
• Build a data warehousing solution using Amazon Redshift
• Build Business Intelligence (BI) reports for historical trending and analysis and enable self-service using Amazon QuickSight

Creating alerts and send notifications for out of bound conditions

An alerting solution is built for monitoring critical paint pretreatment process parameters and KPIs to take necessary corrective actions in near real-time.

Rules are set to examine the streaming values and compare against high and low bounds. Each bound has a warning zone and a critical limit. If a value enters a warning zone and stays in the warning zone for a designated quantity of data points, then an email alert is issued to the Paint Support Team for necessary corrective action.

Visualizing KPIs in near real-time and performing historical analysis

Criteria for data visualization was driven by end user persona needs. The AWS IoT SiteWise Monitor feature is chosen for near real-time operating conditions monitoring for the critical paint pretreatment process parameters and KPIs. Amazon QuickSight is chosen for BI reporting for historical trending and self-service data analysis. These are fully managed AWS services with no infrastructure to manage.

Using AWS IoT SiteWise Monitor, we can explore our library of shop floor assets, and create and share cross-site operational KPI dashboards with Plant Operators. This can be used for near real-time monitoring and visualization of KPIs such as performance, quality, and availability parameters of the overall equipment efficiency (OEE) KPI.

Using the near real-time runtime charts of SiteWise Monitor, the results enable the Genie Operations Team to understand current operating conditions of the paint system. The line graph option is used for most of real-time data visualization needs. This enables the Genie Operations Team to immediately see abnormal conditions and escalate as needed to resolve issues quickly. Engineering can then drill down into the data points to see what other operating conditions are affected.

Amazon QuickSight is used by the Engineering and Quality team members in a much longer historical timeframe with a variety of capabilities to create management reports and further inspect the data.

In addition to the paint pretreatment process data, a one-time historical dataset from Excel was imported into the same data lake in Amazon S3.  Then Amazon Athena is used to access the historical data in Amazon QuickSight for visualizing historical trending. Following is a sample Amazon QuickSight report visualizing historical trending of the pump pressure against its upper and lower operating condition thresholds:

Once the paint pretreatment process data is available in the data lake, the Support and Manufacturing Engineering teams can use Amazon Athena to query the data from the data lake into QuickSight as well as for further on-demand data mining using familiar SQL. Following is a sample dashboard for Stage 1 monitoring of a critical process parameter (pH), which was shared with the team members to view on-demand.

Amazon QuickSight also provides an out of the box Machine Learning (ML) Insights capability with zero coding. Genie is currently exploring this feature to discover hidden trends and outliers, identify key business drivers, and perform powerful what-if analysis and forecasting. This can be achieved with no technical expertise or ML experience.

Repeatable for rapid deployment at scale globally

One of the key principles of the solution was repeatability to deploy in other paint systems globally with minimal changes. Some of the paint systems are similar to the pilot paint system, while the remaining will require some modification to the asset models. The automation is done using the AWS Developer Tools suite following AWS best practices. A step-by-step deployment runbook includes the deployment steps for the pilot plant. Within two weeks of the pilot delivered by the AWS team, the Genie team deployed the automated solution successfully in a new plant paint system. This included hardware setup and new asset model configuration changes.

Key learnings

There were several key learnings from this pilot to highlight. The tight collaboration between Genie IT, OT, and multiple AWS teams helped to move the pilot quickly.

The key factors that contributed to this speed were the following:

• Motivated and nimble one-team mindset of Terex IT, OT, and AWS
• Collaboration tools established early
• High availability of key stakeholders
• AWS IoT SiteWise continued to evolve quickly to meet Genie needs and provided a set of niche features such as asset modeling and SiteWise Monitor that avoided building and maintaining a custom web application for near real-time dashboards
• Access to a low-code / no-code application for the manual data entry could compress the timeline further

Through immersion day labs, training, and practical experience from AWS, the Genie team learned how product, enterprise, and manufacturing data can be connected to deliver new data-driven insights which are either not seen before or have been extremely difficult to get to. From near real-time visualization using SiteWise Monitor to analytics of Amazon QuickSight, Genie team is able to derive the operational insights from the data.

AWS’ customer obsession and working backwards principles delivered tangible results quickly, and established solid procedures for scaling. Knowing that Genie OT team was resource limited, the AWS team deployed a fully automated production ready solution that can be deployed across all the paint systems globally.

Conclusion

In this post, we discussed how Genie, a Terex Brand, is able to ingest, organize, and analyze data from the paint pretreatment process.  The solution helped to identify inconsistent and improper pretreatment process parameters in near real-time. This facilitated the application of necessary corrective actions to improve the downstream paint quality of Genie lifts and hence, maximize customer return on investment.

As always, AWS welcomes feedback. Please submit thoughts or questions in the comments.

Additional Resources

Refer to What is AWS IoT SiteWise? to learn more about AWS IoT SiteWise.

Refer to Collecting, organizing, monitoring, and analyzing industrial data at scale using AWS IoT SiteWise for a multi-part step-by-step walkthrough of a field-to-cloud solution using AWS IoT SiteWise.