Integrate AWS IoT and Esri ArcGIS Velocity for Real-Time Utility Dashboards and Analysis

The Industrial Internet of Things (IIoT) is enabling accelerated digital transformation in many industries, and the power and utilities industry is no exception. Today, IIoT is transforming the way utilities operate to serve customers, optimize operational costs, and enhance the community they serve. One of the cornerstones in enabling this digital transformation is the ability to ingest real-time data from a plethora of industrial sensors that collect data from assets at very high frequency, on the order of millions of events per second. This valuable data, such as vibration frequency in wind turbines, brings new and powerful opportunities to detect issues before they propagate into costly repairs or dangerous operating conditions. Utilities and power generators alike need to access and make sense of the flood of data coming from thousands of assets that comprise their critical infrastructure to better ensure asset health and longevity.

AWS Partner solutions provide services that can natively integrate with AWS IoT and Supervisory Control and Data Acquisition (SCADA) data while providing operational insights. Through the AWS for Industrial initiative, AWS continuously collaborates with many industrial partners to innovate and develop cloud-native solutions for the utility industry.

AWS IoT Core allows utilities to connect IoT devices to the AWS Cloud without the need to provision or manage servers. It can reliably and securely support billions of devices and trillions of messages, process and route those messages to AWS endpoints as well as route to other devices. It is a resilient and scalable mechanism for high-velocity real-time data ingestion, such as from utility assets that can be used to hydrate data lakes built on AWS. This can be taken to the next level by supercharging the real-time telemetry data with geospatial data from geographic information systems (GIS). The merging of real-time telemetry data and geospatial data is a powerful combination that delivers in-depth situation awareness.

AWS partner Esri is a market leader in GIS software, web GIS, and geodatabase management. Esri enables real-time analytic services for IoT data via ArcGIS Velocity, a cloud-native add-on capability for ArcGIS Online. It enables users to ingest data from IoT platforms, including AWS IoT, message brokers, and third-party APIs. ArcGIS real-time capabilities are used across the utility industries by GIS specialists, operations officers, asset management specialists, and engineers working with IoT data. With ArcGIS Velocity, utilities can ingest high-velocity sensor data from AWS IoT into ArcGIS Online to support real-time and big data processing from a geospatial perspective.

This is this Part 2 of a three-part series, in which we explore multiple ways to integrate AWS and Esri products and services and the benefits such integrations provide. In this blog, we explore how to integrate AWS IoT Core and Esri ArcGIS Velocity for Real-Time Utility Dashboards and Analysis. (Part 1 was “Integrate OSIsoft PI and Esri ArcGIS on AWS for Real-time Utility Dashboards and Analysis.”)

It’s worth noting that Esri has two distinct offerings for ingesting real-time sensor data, namely GeoEvent Server and ArcGIS Velocity. This blog will focus on ArcGIS Velocity, which is a cloud-native real-time and big data engine for ArcGIS Online, enabling customers to not only easily visualize IoT data on maps but perform geospatial analysis in real time, near real time, and over time.

Overview of Solution

One of the notable use cases in the renewable energy industry is wind power generation plants, which, if not properly monitored, can lead to catastrophic failures that are expensive to repair and possibly life-threatening. It is critical for plant operators to get an up-to-date overview of the entire plant, as well as all individual turbines and electrical assets within it, in order to optimize power generation and reduce unnecessary capital expenditure and downtime. IoT devices on wind turbines in large wind farms produce lots of data, such as nacelle temperature, generator, gearbox, and blade readings, wind speed and direction, and also high-frequency measurements such as vibration. These critical parameters need to be monitored at each turbine level and as a wind farm collectively to increase power output and enhance the life span of towers and turbines.

To take advantage of the multitude of high-velocity data streams broadcasted by the IoT sensors, utilities need the ability to ingest this data in real-time. As the data comes in, they can trigger alerts or actions and analyze the data for patterns that could provide insights for decision-making. To this end, AWS IoT Core integrated with ArcGIS Velocity provides customers with a wide array of geographically dispersed critical static and mobile assets to have near real-time visibility into their performance. Using ArcGIS Velocity, operators managing wind farms or other energy-generating assets can perform real-time analytics on the streaming data. This provides immediate insights to detect hidden problems and predict failures before they occur.

Figure 1 illustrates these overall concepts. Real-time IoT data from wind turbines is ingested securely and reliably at scale using AWS IoT Core and then to Kafka. A feed service running in Velocity then ingests the data from Kafka into the ArcGIS platform. Velocity supports a wide range of services and protocols. AWS IoT Core is a great way to get up and running streaming data into Velocity. Kafka was chosen for this proof-of-concept because of its additional data aggregation capabilities as well as the benefit of bidirectional support in Velocity.

Figure 1: Integration Concept Overview

Integrated AWS IoT and Esri Velocity Reference Architecture

The reference architecture shown in Figure 2 lays out the data flow in the next level of detail. Customers can ingest a variety of industrial protocols by using applications like KepServerEx at the edge for industrial protocol conversion and AWS IoT Greengrass for edge processing to filter noisy data. AWS IoT Core and AWS IoT rules can be used to aggregate and filter sensor data with advanced mathematical functions, which then send the data to Apache Kafka running on EC2 instances in order for ArcGIS Velocity to ingest data via a Kafka feed connector.

Figure 2: Reference Architecture Data Ingestion from Assets to ArcGIS Velocity via AWS IoT Core

1. Real-time data from critical assets is ingested via a KepwareEX or a similar server running at the edge.

1. • KEPServerEx supports multiple input protocols and connects to AWS IoT Greengrass using MQTT messages.
   • AWS IoT Greengrass is an open-source Internet of Things (IoT) edge runtime and cloud service that helps you build, deploy, and manage IoT applications on your devices. AWS IoT Greengrass enables your devices to collect and analyze data closer to where that data is generated, react autonomously to local events, and communicate securely with other devices on the local network.
   • Secure communication between AWS IoT Greengrass and KepwareEX is enabled by KepwareEX verifying the device certificate from AWS IoT Greengrass.

2. AWS IoT Greengrass edge devices communicate securely with AWS IoT Core, streaming their near real-time IoT data to the AWS Cloud. Once the data is in AWS IoT Core, the messages can be consumed by subscribing to a topic.

3. IoT Core can integrate with Kafka via rules. You can configure rules and craft policies to filter and normalize data before sending it to Kafka. Currently supported mechanisms are SSL and SASL_SSL.

4. ArcGIS Velocity subscribes to Kafka via SASL_Plaintext or Plaintext and will receive data from AWS IoT Core through Kafka in real time. Additional filtering and aggregation can be done in Kafka before the data is forwarded to ArcGIS Velocity. When configuring the Kafka connection in ArcGIS Velocity, you can also customize the schema as shown below.

Figure 3: ArcGIS Velocity will receive data from AWS IoT Core through Kafka in real time.

5. As the data is received by the feed in ArcGIS Velocity, you can perform geospatial analytics and mapping, for example, monitoring turbine readings relative to nearby sources of environmental risk, such as storm fronts or lightning strikes.

Using ArcGIS real-time analytics, hazards such as lightning strikes within a given distance of turbines can be configured to trigger alerts to operator personnel for follow-up action. Moreover, these kinds of incidents can be elevated to also trigger an actuation event under a defined set of conditions, for example, triggering a shut-down of the turbine if a fault error code suddenly begins being reported within a given amount of time following the nearby lightning strike event.Actuation needs, based on this synthesis of sensor readings and geospatial event information, can be fulfilled by sending incidents or commands back to Kafka and AWS IoT Core from ArcGIS Velocity.

Real-time geospatial data stored to ArcGIS over time can also be used to drive strategic insights for decision-making, especially in the area of condition-based maintenance. In the example of weather hazards in and around wind power generation farms, event history (like lightning strikes) can be summarized spatially and temporally to illuminate which turbines have been subject to the most activity. This information can then be used to inform inspection frequency for field crews or drones, increasing the likelihood of identifying problems before they propagate into more expensive repairs.

Conclusion

As outlined in this blog, the ability to stream, ingest, and leverage high-velocity IoT data from critical static and mobile assets is essential. It provides power and utilities industry operators near real-time visibility into their enterprise. With data ingested securely and at scale using AWS IoT and Esri’s ArcGIS Velocity, utilities of all sizes can perform real-time analytics on the streaming data from critical assets. They can gain visibility into network issues, monitor the location of field crews, remediate faults, and perform proactive maintenance to improve asset efficiency and reduce operational costs.