Real-time operational monitoring of renewable energy assets with AWS IoT

IoT technology has led to an explosion of data being ingested by business value streams. Manufacturing, power and utilities, telecommunications, transportation, and retail are part of a growing group of industrial markets that now realize the far reaching impact of IoT data and the tremendous value that can be derived from it.

Consider European utilities that are undergoing a smart meter rollout. In 2021, is it expected that over 34 million smart meters will be installed across European utilities. A single smart meter at a modest collection interval of 15 minutes can generate 96 meter readings per day and even more as we increase the interval frequency. This means that there will be an exponential growth in the quantity of meter readings and in the utility’s potential to deliver knowledge value from the metering data stream deluge. Likewise, a single weather station can generate hundreds of atmospheric parameters every day. Across our cities today, there are thousands of IoT sensors in solar panels, parking lots, public transportation systems, and even in our homes. These all generate massive amounts of telemetry data that can be harnessed for societal gains and benefit to consumers.

However, utility, manufacturing, and government organizations are not investing in IoT platforms for just the sake of technology. Rather, they are procuring them for delivery of specific industry line of business (LOBs) value outcomes. To that end, organizations need to have a solutions value chain in place to ingest this high velocity data, store it securely, enrich it, run analytics on it, visualize it in business-friendly formats, and finally, derive value from it. Here is where AWS IoT comes in. AWS IoT provides a complete set of managed services encompassing:

• Device software
• Connectivity and control services
• Security services
• Analytic services

Furthermore, AWS IoT natively integrates with AWS storage, streaming, and compute services, such as Amazon S3, Amazon Kinesis, AWS Identity and Access Management (IAM), Amazon RDS, AWS Lambda, Amazon SNS, Amazon SQS, and more. IoT solutions are complex and multidimensional. AWS IoT removes this complexity by providing a set of managed services that carry out undifferentiated heavy lifting, allowing organizations to do what they do best: looking for places to apply IoT to innovate and solve LOB problems strategic to their organization. AWS IoT emancipates information technology organizations from managing their IoT infrastructure and empowers them with the intelligence to achieve greater operational efficiency and revenue growth, outcomes key to their respective lines of business.

For example, Philips HealthSuite Digital Platform (HSDP) leverages AWS IoT services to manage, update, remotely monitor, and collect data from smart devices. Another example is Wärtsilä, a global leader in smart technologies and complete lifecycle solutions for the marine and energy markets. Using AWS IoT to ingest GPS data and route it to Amazon ML, where new models for predicting engine failure were constructed, Wärtsilä is now equipped to proactively predict engine failure with greater than 90% accuracy.

Realizing real-time dashboards with AWS IoT

A somewhat underestimated aspect of this value chain is data visualization. Irrespective of industry, data visualization is critical in delivering value from curated data lakes that have been hydrated by IoT, social media, industrial, telemetry, and other high velocity data streams.

A real-time, low latency interactive dashboard of events conveys situational awareness in the most efficient way, allowing customers to gain agility from timely knowledge and to quickly take business actions. At the same time, it provides an effective mechanism for transmission of meaningful insights. Live dashboards allow organizations to recognize patterns in data, correlate them to events, and determine a response strategy that enables quick and decisive decision making. An AWS IoT-powered live dashboard is a resilient collaboration of cloud technology and real-time situation awareness.

We explore this further in the following reference architecture:

In this scenario, we have wind turbines, weather stations, and solar panels generating telemetry data and sending it to AWS IoT Core. This reference architecture achieves the following objectives:

• Scalable and secure high velocity data ingestion
• Scalable and real-time situational awareness serverless dashboard
• Hydration and curation of IoT data lake
• BI reporting
• Alarm management

The data flow is further described in the following:

1. Data ingestion

The sensors’ PLCs send live data over message queuing telemetry transport (MQTT), after possible conversion from OPC-UA, DNP3, Modbus,  protocol at a predetermined interval frequency (10 seconds, 15 minutes, etc.) to the AWS IoT Core via a pub-sub mechanism. The collection frequency can be different for each sensor. All MQTT messages are read by the IoT rules engine. At this point, there are two alternatives on how the streaming data can be ingested. They are described in the following:

• The AWS IoT rules engine directs all MQTT messages to Amazon Simple Queue Service (Amazon SQS). Amazon SQS is a fully managed message queuing service that allows us to decouple and scale microservices, distributed systems, and serverless applications. MQTT messages are typically small in size, in the range of kilobytes (kb) that can fit well in a queueing mechanism like Amazon SQS. With Amazon SQS, we can transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. Amazon SQS stores copies of every message redundantly across multiple Availability Zones so that they are available whenever needed. With a pay-as-you-go model, Amazon SQS provides significant cost saving versus the “always-on” model of self-managed messaging middleware. Also, it comes with native integration with Amazon S3. In this option A, Amazon SQS deposits all MQTT messages in the Amazon S3 landing zone.
• The AWS IoT rules engine directs all MQTT Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is a scalable and durable real-time data streaming service. Kinesis Data Streams can continuously capture gigabytes of data per second from hundreds of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. In this option, Kinesis Data Streams forwards all MQTT messages to Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose allows us to reliably load streaming data into data lakes, data stores, and analytics services such as Amazon S3, Amazon Redshift, and Amazon OpenSearch Service (September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service). In this option B, Amazon Kinesis Data Firehose deposits all MQTT messages in the Amazon S3 landing zone.

Both of the preceding options are fully managed, reliable, and scalable, but each have different price/performance curves. Every customer’s use case will be different. It is recommended to carry out a comparative testing of both the mechanisms to get a clearer understanding of total cost and latency when deciding which option suits your needs best.

2. Grafana live dashboard

Both options make use of AWS Lambda to write real-time telemetry data to Amazon Aurora Serverless, which acts as the data source for Grafana. The process is described in the following:

• AWS Lambda can process messages both from Amazon Kinesis Streams and Amazon SQS. The Lambda in the architecture earlier in this post utilizes its batch window functionality, which allows it to wait till the payload reaches an appropriate size. At that point, the Lambda is invoked. The batch operation increases the average number of records passed to the function with each invocation. This helps to reduce the number of invocations and optimize cost.
• When invoked, the Lambda writes the payload data into an Amazon Aurora Serverless RDS instance. Amazon Aurora is a serverless, MySQL, and PostgreSQL-compatible relational database. It is an on-demand, automatic scaling database that will automatically start up, shut down, and scale capacity up or down based on your application’s needs. It enables you to run your database in the cloud without managing any database instances.
• Amazon Aurora Serverless acts as the data source of the Grafana server running on AWS Fargate. AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS). Fargate removes the need to manage any servers and you only pay for the resources to run the container. There is no over or under provisioning. Grafana on Fargate securely connects to Amazon Aurora Serverless as a data source to publish real-time dashboards.

3. AWS IoT data lake

• The IoT Amazon S3 data lake can be hydrated by Amazon SQS or Kinesis as mentioned in step 1 earlier. Either service can deposit each MQTT message as a JSON object in an S3 bucket, serving as a landing zone. The landing zone is partitioned in the order of year-month-day and is continuously being hydrated in real time.
• AWS Glue is triggered at a periodical interval (for example, every 24 hours, etc.) to transform all the daily JSON files into a columnar parquet format. This makes it more consumable for downstream applications.
• The parquet files are stored in another Amazon S3 bucket that serves as our business zone. The business zone is partitioned in the order of year-month-day.

4. BI reporting

• Amazon Redshift is a fast, scalable data warehouse service that can perform complex queries on massive collections of structured data with super-fast performance. It comes with native integration with Amazon S3 and with Redshift Spectrum we can directly query on exabytes of unstructured data in Amazon S3 without having to load or transform any data. Furthermore with Amazon Redshift Federated Query, we can query across Amazon Redshift, Amazon S3, and other relational database services, such as Aurora Postgres. This makes Amazon Redshift the data warehouse of choice for powering BI portals and reports.
• Amazon QuickSight is a fast fully managed BI service that comes with out of the box integration with Amazon Redshift. Amazon QuickSight BI reports can be accessed from any device, and embedded into your applications, portals, and websites. Amazon QuickSight’s in-memory calculation engine SPICE allows us to achieve superfast performance at scale. Also with Amazon QuickSight’s pay-per-session pricing model, you only pay when users access the BI reports.

5. Alarm management

Renewable energy assets such as wind turbines and solar panels are expensive pieces of technology. On-time alerting in response to incidents, events, and anomalies is crucial to ensure operational efficiency. This is where we utilize AWS IoT Events. AWS IoT Events is a managed service that continuously monitors data from equipment to identify their state, detect changes, and trigger the appropriate responses when changes occur. An operational event can be described as any of the following:

• Equipment malfunction
• Any measure of poor quality
• Signs of wear and tear on equipment requiring preemptive maintenance
• State of a process

AWS IoT Events helps us address two common problems:

1. Predict, prevent, and diagnose
   • Predictive maintenance
   • Predictive quality (detect and eliminate potential quality issues at their source)
   • Accurate failure diagnosis (reducing false alarms)
2. Improve efficiency
   • Improve the efficiency of processes, products, equipment, and staff
   • Increase throughput and speed

In light of the preceding, consider the following examples with regard to the previously mentioned reference architecture.

• Micropitting – The gearbox and bearings are two components of a wind turbine that are most prone to failure. A continuous high gust of wind at an incorrect pitch can misalign the main shaft and gearbox. This can lead to excessive vibration of the internal assembly and micropitting of the gearbox roller surface. AWS IoT Events detector models can continuously monitor the state of wind speed, wind direction, angle of attack, rotor speed, noise, and vibration. When the vibrations exceed the set threshold limits, AWS IoT Events invokes Amazon Simple Notification Service (Amazon SNS) to send out an alert to maintenance staff via SMS or email. They can then take pre-emptive corrective action to prevent any further damage such as apply brakes on the turbine.
• Fire – A buildup of contaminants in the lubrication system coupled with an increasing ambient temperature can quickly escalate to a mechanical fire. AWS IoT Events detector models can continuously monitor the nacelle temperature, humidity, and air quality. When the detector determines that the nacelle temperature has been continuously beyond the acceptable thresholds, say for example greater than 40 degrees Celsius for more than three hours, it issues as alert via Amazon SNS to maintenance staff via SMS or email. It is also possible to communicate with other AWS IoT services like device shadow and AWS IoT Greengrass, which can be configured to apply emergency brakes to shut down the turbine in response to severe incidents.

Sample Grafana dashboard

The following are a few screenshots of an AWS IoT-powered renewable asset (wind farm) dashboard, as described in the previous reference architecture:

Conclusion

This reference architecture demonstrates how AWS IoT services can be used to achieve a highly available and scalable, real-time operational monitoring system for renewable energy assets, such as wind turbines and solar panels. By utilizing the gamut of AWS IoT services and best practice cloud technologies, we can efficiently serve the needs of operation/maintenance supervisors, data engineers, and data scientists, and provide real-time situational awareness of renewable energy generation assets. For more information, visit AWS for Power and Utilities.