AWS Batch enables near-real-time energy production forecasts using NVIDIA Earth-2

This post was contributed by Satheesh Maheswaran, Vidyasagar Ananthan, Timothy Brown, Srinivas Tadepalli, Colin Bridger, and Ryan Qi from AWS, and Stefan Weissenberger and Farah Hariri from NVIDIA

In this post we’ll show how you can run and scale weather forecasting models based on machine-learning (ML) algorithms and combine them with conventional physics-based wind farm simulations to forecast the energy production of a wind farm.

This not only provides a rapid energy calculation for a given wind farm in a timely manner but also demonstrates that weather predictions are possible using ML methods coupled with downstream applications like wind farm simulations.

To pull this off, we use AWS Batch with the NVIDIA Earth-2 platform to explore artificial intelligence (AI)-driven meteorology and employ the open-source TwinGraph orchestration library. These can all be combined into a workflow on AWS that allows scientists and engineers to explore potentially millions of possible scenarios at a fraction of the cost of traditional numerical weather simulations.

Background

Both traditional weather forecasting organizations and technology companies are evolving their approach to forecasting. Machine learning approaches hold significant promise.

Figure 1 – forecasting weather with any level of accuracy is a computationally intense exercise. New techniques drawn from deep-learning provide hope that these simulations can be sped up dramatically. This figure shows the ensemble mean of 10meter zonal winds around the globe

Currently, we’re seeing the impact of these models as a viable tool in combination with traditional numerical weather prediction (NWP) models to perform downstream tasks: like electricity load demand forecasting or wind energy production. In most cases, they’re reducing time to solution and costs.

In this post, we’ll show how AI approaches to weather prediction can be used in tandem with renewable energy production models, specifically to forecast the energy output over the course of a week. This approach is a viable and cost-effective option compared to traditional NWP solutions and can help scientists and engineers explore parameter space more rapidly.

Why are rapid tools for energy forecasting important?

Global medium-range weather forecasting plays a vital role in decisions that impact our everyday lives. Historically, NWP uses high performance computing (HPC) to solve equations that govern the physics of weather. Calculations like this are a costly and time-consuming activity and are typically repeated several times a day or even every hour. Forecasters initialize the numerical models with observation and measurement data to predict weather forecasts.

NWP methods have evolved consistently, gradually improving the accuracy of weather forecasts, but there’s an element of uncertainty in all predictions including weather predictions. This is because weather is a chaotic dynamical system, highly sensitive to initial conditions.

For medium-range forecasts, two factors can lead to a decrease in accuracy as the forecast lead-time increases:

1. Uncertainty in initial conditions: measurement data for current weather conditions has an element of error and there are large regions without accurate monitoring stations.
2. Approximation error inherent in the model assumptions: numerical weather models (whether physics-based or data-driven) include an element of error due to system approximations – for example by not capturing the physics and spatial-temporal scales necessary to encapsulate the system behavior.

These uncertainties amplify as the forecast window increases. Quantifying uncertainty in weather forecasting is extremely important given the potential it offers to minimize the socio-economic impact of extreme weather patterns through better planning. One way to reduce the uncertainty is to combine the results of multiple forecasts – a technique known as ensembles.

Running ensemble forecasts usually means running additional NWP calculations at an increased cost. Large ensembles with 100s to 1000s of members using NWP models bring modern supercomputers to their limits. However, ML weather models, like those available through Earth2Studio, make it possible to run ensembles on a small number of NVIDIA GPUs. Using AWS services, we will show you how to build a single workflow to integrate ensemble weather forecasts with downstream applications.

Figure 2 – A 100-ensemble forecast of surface windspeed (eastward (u10m) and (northward) v10m component). Figure shows the standard deviation across a 7-day forecast demonstrating the uncertainty in forecasting using FourCastNet with Gaussian noise distribution

How weather forecasts can use machine learning

Recent advances in machine learning for modeling various physical phenomena (including weather forecasting) have provided a new option to augment the traditional approach of NWP.

The European Centre for Medium-Range Weather Forecasts (ECMWF) has made the atmospheric re-analysis of the global data (ERA5) available from 1940 to present. Consequently, the community has been able to create multiple ML models with benchmarks available via WeatherBench.

In this post, we will couple FourCastNet (available through Earth2Studio from NVIDIA) with a medium-fidelity wind farm simulation model available in PyWake. NVIDIA Earth-2 is a full-stack, open platform that accelerates climate and weather predictions with interactive, AI-augmented, high-resolution simulations. FourCastNet provides a global medium range prediction at 0.25-degree resolution.

NVIDIA trained the model on a subset of ERA5 reanalysis data gathered from 1980 to 2015. FourCastNet inferencing can be completed in seconds on a single GPU, orders of magnitude faster than traditional NWP simulations, thereby reducing cost and time to solution. Specifically, this allows researchers to run a significant number of ensemble calculations to reduce the uncertainty and combine with NWP simulations to further reduce the likelihood of error. This ultimately improves medium-range probabilistic forecasting and augments existing NWP models. The Earth-2 platform provides forecast models like FourCastNet, GraphCast, DLWP and easy-to-use workflows to apply these models for specific use cases.

Wind farm modeling application

The windIO framework was created by the International Energy Agency (IEA) “wind task 37” team and defines the inputs and outputs for systems engineering multidisciplinary design analysis and optimization (MDAO) of wind turbines and plants. The benchmark provides an industry-standard open-sourced layout to develop best practices in system design and help with designing tools and methods related to research in wind energy. The workflow discussed in this post can be applied to any wind farm design choice that the reader may wish to experiment with, but the results presented here are from the open-source model available in IEA-37.

To calculate the energy output of the IEA-37 windfarm benchmark, an initial windspeed is necessary. This windspeed was calculated by inference from the NVIDIA Earth-2 platform. The energy of the windfarm was calculated by using the analytical wake model with turbulence that’s part of the open-source PyWake project. For this post, we simulated passive yaw control without consideration of wake steering and associated phenomena. The workflow we’re presenting here is agnostic to the underlying simulation environment. This will enable you to experiment with alternative methodologies, supported by AWS infrastructure that will elastically scale to meet your computing requirements.

The tools provided by AWS and NVIDIA enable the rapid forecasting of energy production, opening new ways of approaching this challenge. We can – for example – integrate AI weather predictions models together with well-known energy production modeling tools such as PyWake into a single workflow. The results of this coupling can aid in planning the power grid optimization strategies at local and national levels, both during the design process and in production to calculate energy output.

Coupling ML weather prediction with wind farm energy output

Earth2Studio provides pretrained models and workflows to execute AI weather forecasts as part of the NVIDIA Earth-2 platform. FourCastNet, one of the models supported by Earth2Studio, predicts wind speed and other meteorological variables like geopotential height, temperature, and humidity. Using FourCastNet in the ensemble workflow implemented in Earth2Studio, we can run multiple inferences to get a statistical distribution for the required set of variables.

The inference pipeline requires initial conditions and Earth2Studio supports multiple data sources for initial conditions. For this post we used the Global Forecast System (GFS) model by NOAA for our initial conditions. The GFS data sets are available as part of the Registry of Open Data on AWS. We use the initial conditions from the GFS model to initialize the inference model of FourCastNet.

Inferences are calculated using the FourCastNet model for five different wind farm locations using the assumption that each wind farm contains 64 turbines. We used the longitude and latitude of the windfarm locations as the input parameters for forecasting the windspeed. We forecasted the windspeed across a 7.5-day period (this implies 30 inferences at six-hour intervals). We repeated the inferences for 1,000 ensemble runs at each location by adding Gaussian random field for spatially uncorrelated noise – one of the several noise-generation schemes offered by Earth2Studio.

We ran a total of 150,000 inferences for five different wind farm locations using 25 NVIDIA A10G Tensor Core GPUs in well under an hour. This is significantly faster and cheaper than running NWP simulation and scaled to many more thousands of inferences through AWS Batch with ease. We extracted the u100m and v100m variables from the inferences. u100m and v100m refer to eastward and northward components of the wind respectfully at 100m height – these allow for the calculation of wind magnitude at a given direction and time.

The outputs of the inference from FourCastNet, specifically the windspeed, are then used as input to PyWake to calculate the energy of a given wind farm, with a further subdivision in time to 36-minute intervals using Weibull sampling, to account for variations in wind velocity across a 6-hour period.

Infrastructure

We used Amazon Elastic Compute Cloud (Amazon EC2) instances, and scaled these using AWS Batch. Specifically, we used g5.4xlarge instances because the instances have 16GB of GPU memory that is sufficient to run the inference, and this provides a balance between sufficient computing power and a cost-effective solution.

Figure 3 shows the infrastructure we deployed to perform the inference calculations.

Figure 3 An architecture diagram to couple ML inferences running on AWS EC2 GPU instances powered by NVIDIA A10G Tensor Core GPUs through AWS Batch and coupled with CPU architectures for wind-farm simulations. The workflow is orchestrated by the TwinGraph library.

Results

Predicting energy output using PyWake requires windspeed at a given location and height. Traditionally, NWP simulations calculated this.

Figure 4 shows the forecasted total energy production for 5 offshore wind farms. The forecast is for a 7.5-day period, initialized with data from the GFS model on the 20^th of May 2024. We used the FourCastNet model to generate an ensemble of wind speeds by adding a Gaussian random field for spatially uncorrelated noise.

The weather system is representative of a chaotic dynamical system where small changes to initial conditions result in significant deviation over time, and you can see this in Figure 4 with the energy calculations as they progress.

Figure 4 – Energy production of 5 wind farms across the globe calculated for 7.5 days. Each line corresponds to one inference run for wind speed and used as the input for wind energy calculations using PyWake.

Benefits for users

We showed you in this post how you can combine ML weather inferences with traditional simulations such as wind farm energy production in a cost-efficient and rapid manner. We showed how you can integrate the entire hypothesis testing into a single event-driven workflow thanks to AWS Batch, TwinGraph, and NVIDIA Earth-2.

We could improve prediction accuracies by incorporating other ML models for weather forecasting and higher fidelity windfarm simulations. Thanks to the efficient tooling, users can incorporate their own AI weather prediction models and simulation frameworks. The workflow can also be extended to predict the supply and demand for resources for a longer range – such as 14 days.

NVIDIA also plans to extend Earth-2 utilities and optimized pipelines for scaling the workflows on NVIDIA GPUs and to make them available for enterprise production use under the NVIDIA AI Enterprise software platform, available in AWS Marketplace.

This approach still requires further development to be applicable to industry in a production setting, but demonstrates the potential of integrating ML techniques for weather with physics-based simulations.