Deploying a High Performance Computing solution for accurate weather and renewable energy production predictions

The field of meteorology has long relied on computational models to predict weather patterns. The advent of High-Performance Computing (HPC) and machine learning (ML) mean that these predictions have become more accurate and reliable. This post outlines the steps for deploying an HPC cluster for weather forecasting on the Amazon Web Services (AWS) Cloud. We examine how cloud-based solutions provide computational flexibility and scalability. Our goal is to demonstrate how HPC and cloud technology help Iberdrola’s meteorologists calculate asset production forecasts.

For Iberdrola, with its renewable energy production of 80.3 TWh in 2024, weather forecasting is crucial. Spain’s March 2004 regulatory framework for renewable energy production established requirements for production forecasting. This led Iberdrola Renovables to develop MeteoFlow, an in-house forecasting tool with a 20-year track record.

Throughout its evolution, MeteoFlow has incorporated advanced meteorological forecasting techniques, ML, AI, and big data technologies. Experts in various fields enabled these advances through their system maintenance, monitoring, and development.

MeteoFlow estimates production outputs for Iberdrola Group’s renewable installations in the coming hours and days. This strategic information enables optimal decision-making in energy markets and helps identify time windows for scheduling Operation and Maintenance (O&M) activities with minimal disruption.

Through extensive technical expertise, in-depth knowledge of renewable energy technologies, and seamless integration with operational tools, MeteoFlow achieves accuracy rates exceeding market standards. Using meteorological data—primarily wind forecasts for wind farms and solar radiation forecasts for photovoltaic plants—MeteoFlow generates production forecasts for all renewable facilities, such as onshore and offshore wind farms, photovoltaic plants, and hydroelectric installations. The system serves all countries where Iberdrola operates, customizing forecasts to meet local requirements and user needs.

MeteoFlow combines global-scale numerical simulations from leading meteorological centers with internally operated regional models specifically configured for Iberdrola’s worldwide installations. These models process Navier–Stokes equations and need substantial computational power to handle the complex nonlinear partial differential equations that perform millions of calculations per time step.

Figure 1. Navier–Stokes Equations

The computational power needed to generate a forecast, whether through analytical approaches or ML techniques, is significant. The data structures used as input are complex and come in high volumes, needing technology that can provide both nearly unlimited storage capacity and very high I/O throughput. These two requirements—processing large volumes of data at scale and accessing sufficient compute capacity—make the cloud an ideal platform for running our forecasting models.

Business value

The MeteoFlow system offers substantial business value through its advanced forecasting capabilities and integration of cutting-edge technologies. The program refines numerical forecast data using the latest ML and AI techniques to provide hourly forecasts up to 96 hours ahead and daily forecasts up to 10 days in advance for more than 450 wind and photovoltaic plants. This precision is crucial because all generation technologies, such as renewables, must predict their production for electricity markets hours or days in advance, and face penalties for inaccurate forecasts. More accurate energy production predictions lead to higher revenues, improved profitability, and increased growth capacity for renewables when compared to other energy sources.

Since 2016, MeteoFlow has provided long- and short-term meteorological prediction maps on Iberdrola’s intranet. For the specialized field of renewables, it includes an integrated tool within Domina—the suite of tools for the operation and maintenance of installations—which provides users with direct access to prediction maps and weather and power forecasts tailored to each installation in Europe, the United States, Mexico, Brazil, and offshore wind farms. This system is complemented by weather alerts for conditions affecting installations, such as extreme winds, storms, and frost. The availability of this information enhances the work of various roles in the company, including O&M planning, asset management, and meteorological risk assessment.

Flow prediction is another valuable MeteoFlow functionality that is crucial for forecasting potential energy production in our hydroelectric installations and supporting hydraulic operations, maintenance, and energy management. Using available meteorological observations, the tool generates forecasts for up to 10 days.

Wave prediction is essential for our offshore installations, such as wind turbines, marine wind farm components, and emerging floating photovoltaic systems. These predictions, which include wave height data, optimize operations and maintenance schedules. MeteoFlow offers both short-term hourly wave predictions for up to 72 hours and long-term forecasts extending to 10 days.

MeteoFlow provides visualization and analysis tools through MeteoWeb, which is accessible on demand to Iberdrola users, to optimize the extensive data stored in the system. These analytical capabilities support multiple functions: operational and maintenance planning, energy management, performance analysis, and the development of new prediction methods by the MeteoFlow team. This comprehensive approach allows MeteoFlow to continue enhancing renewable energy production and strengthening Iberdrola’s position in the energy market.

Figure 2. Screenshots of MeteoWeb for East Anglia I Offshore Wind Farm, the 4th of April 2025

Solution architecture for MeteoFlow at AWS

We defined three infrastructure requirements to guide the solution design, as shown in the following figure:

1. Handling large volumes of data without compromising performance or quality. We needed to migrate 300+ TB of data from our on-premises location to the AWS Cloud.

2. Real-time forecasting. Our solution must perform data processing and analysis in real time, enabling decisions based on up-to-date and relevant information.

3. Technology integration and seamless collaboration. We needed a system that would facilitate third-party integration through a modular and flexible architecture. We also needed it to be designed to interoperate at both technical and team levels, allowing our organization to benefit from our predictive analytics.

Figure 3. The architecture diagram of our system after being migrated to the cloud

The solution uses several AWS foundational services. Amazon S3 stores data from multiple sources, including global and regional models such as IFS, GFS, ICON-EU, and our proprietary models. Amazon S3 provides industry-leading scalability, data availability, security, and performance.

For computation, we have used Amazon Elastic Compute Cloud (Amazon EC2) HPC instances in the Ireland AWS Region. The hpc7a.96xlarge instances, featuring 192 physical cores, are a good fit for this workload. These instances run on 4th generation AMD EPYC processors (AMD EPYC 9R14) with 768 GiB of memory. They include Elastic Fabric Adapter (EFA) support for network speeds up to 300 Gbps and are powered by the AWS Nitro System—a combination of dedicated hardware and lightweight hypervisor.

Amazon Elastic File System (Amazon EFS) provides serverless, fully elastic file storage to store and share computation results across processes.

Amazon SageMaker provides integrated experience for analytics and AI with unified data access. You can train ML models in SageMaker to achieve higher forecast precision.

To migrate 300+ TB of meteorological prediction data from our on-premises location to AWS, we initially considered using AWS Snowball. Snowball is a physical data transfer service that helps you migrate large amounts of data to and from AWS. However, our analysis revealed that transferring data to the Snowball device through our local area network (LAN) would operate at the same bandwidth as our existing connection to the Europe (Ireland) Region due to the high-speed communication link we have between our on-premises data center and the AWS Cloud.

Instead, we implemented AWS DataSync, a service that optimizes and secures data transfer between on-premises storage and Amazon S3. The following figure shows this architecture.

Figure 4. The configuration used to move the data from on-premises to Amazon S3

The high-level steps needed to set up this architecture are:

• Establish an AWS Direct Connect connection between your on-premises network and the AWS Cloud.
• Configure a Virtual Private Gateway (VGW) in your Amazon Virtual Private Cloud (Amazon VPC).
• Create a Direct Connect Gateway and associate it with your VGW.
• Set up a Private Virtual Interface for your Direct Connect connection.
• Associate the Private VIF with your Direct Connect Gateway.
• Configure your on-premises router to route traffic to AWS through the Direct Connect connection.
• Create a DataSync agent in your on-premises environment.
• Activate the agent using the activation key provided by AWS.
• Create a source location for your on-premises storage.
• Create an Amazon S3 destination location in AWS.
• Specify the source and destination locations.
• Configure a DataSync task and execute the task.

This setup enabled us to move all of the MeteoFlow historical data to the AWS Cloud.

Conclusions

The movement of MeteoFlow to the AWS Cloud marks another milestone in the already long history of MeteoFlow. AWS offers MeteoFlow the possibility to use elastic compute and storage capacity to store very large volumes of data in a managed service mode and the possibility to use services such as Amazon SageMaker. All of this helps us to focus on the development and deployment of our ML workloads.