Skip to main content

AWS Solutions Library

Guidance for Building a High-Performance Numerical Weather Prediction System on AWS

Open guide
Go to sample code

Overview

This Guidance shows how to predict the weather over the Continental United States (CONUS) by deploying the Weather Research and Forecasting (WRF) model on AWS. Provided by the National Center for Atmospheric Research (NCAR), the WRF model helps support atmospheric research and operational forecasting applications. By running the WRF model using high performance computing (HPC) clusters on AWS, you can maximize the performance of your weather prediction workloads to accurately and reliably predict, plan, and manage weather forecasts.

How it works

HPC Cluster Deployment

This architecture diagram shows how to provision the AWS ParallelCluster user interface (UI) and configure an HPC cluster with compute and storage capabilities. For the numerical weather prediction workflow, open the other tab.

Download the architecture diagram
Architecture diagram showing an AWS High Performance Computing (HPC) solution for numerical weather prediction. The workflow includes AWS CloudFormation, Amazon API Gateway, AWS Lambda, authentication and authorization with Amazon Cognito, Session Manager, NICE DCV, AWS ParallelCluster managing head and compute nodes, and Amazon FSx for Lustre, all within a virtual private cloud (VPC) in the AWS Cloud.

Prediction Workflow

This architecture diagram shows how to predict the weather for CONUS by deploying the WRF model on AWS and setting up the numerical weather prediction workflow. For the HPC cluster deployment, open the other tab.

Download the architecture diagram
Architecture diagram illustrating the workflow for numerical weather prediction on AWS, featuring AWS ParallelCluster, NICE DCV for visualization, Slurm Workload Manager, Amazon FSx for Lustre, and Amazon EC2 Auto Scaling for HPC workloads.

Well-Architected Pillars

The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.

This Guidance uses a combination of fully managed services (including API Gateway, Amazon Cognito, and Lambda) and self-managed services (including FSx for Lustre and Amazon EC2). The latter services are deployed to a configurable HPC cluster by means of a template and can be reconfigured or updated if cluster performance requirements change. You can use Amazon CloudWatch to monitor all these services through event logging.

Read the Operational Excellence whitepaper 

Amazon Cognito and API Gateway provide secure authentication and authorization and secure API access management. You can then log in to the HPC cluster’s head node for application deployment and management, using the AWS Systems Manager Session Manager secure shell—which provides greater security—or by using NICE DCV. Additionally, FSx for Lustre provides data encryption both in transit and at rest. By scoping AWS Identity and Access Management (IAM) policies to the minimum permissions required, you can limit unauthorized access to resources.

Read the Security whitepaper 

AWS ParallelCluster uses HPC cluster job scheduling to enable parallel computational task implementation, using Slurm Workload Manager, which optimally allocates resources based on job requirements, priorities, and user-defined policies. This reduces the chance of application failure so that you can run weather simulations and avoid downtime errors. Additionally, this Guidance deploys EC2 instances in different Availability Zones for increased reliability, and FSx for Lustre provides highly reliable storage for your HPC clusters.

Read the Reliability whitepaper 

This Guidance lets you efficiently manage and provision HPC clusters using AWS ParallelCluster and a YAML-based configuration. AWS ParallelCluster efficiently scales its CPU and RAM footprint and the instance number both horizontally and vertically to handle increased workloads. This Guidance also uses Message Passing Interface to provide efficient parallel processing and distributed data processing capabilities. Additionally, FSx for Lustre provides a high-performance storage layer for the HPC clusters.

Read the Performance Efficiency whitepaper 

As a managed service, Amazon Cognito provides cost-effective user authentication and authorization. Additionally, Amazon EC2 Auto Scaling scales cluster node instances horizontally or vertically based on workload demand, so that you won’t have to provision and pay for unused resources. FSx for Lustre also provides a cost-efficient storage layer that makes it easy to launch, run, and scale storage for your HPC tasks.

Read the Cost Optimization whitepaper 

This Guidance uses specialized Amazon EC2 instances (including Hpc6a instances powered by third-generation AMD Epyc processors) that offer high performance for compute-intensive HPC workloads. This performance, combined with the elasticity and scalability of serverless of AWS services, helps you achieve optimal resource utilization, helping you avoid overprovisioning resources. Additionally, FSx for Lustre supports concurrent access to the same files and directories from thousands of compute instances, further helping you minimize your workloads’ environmental impact.

Read the Sustainability whitepaper 

Deploy with confidence

Everything you need to launch this Guidance in your account is right here

We'll walk you through it

Dive deep into the implementation guide for additional customization options and service configurations to tailor to your specific needs.

Open guide

Let's make it happen

Ready to deploy? Review the sample code on GitHub for detailed deployment instructions to deploy as-is or customize to fit your needs. 

Go to sample code

Disclaimer

The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.