Building the WIS 2.0 global weather cache on AWS

The World Meteorological Organization (WMO) wants to build and modernize a global weather framework with WMO Information Systems (WIS) 2.0 to enable and democratize unified access to critical, up-to-date weather data across the world. Currently, lack of observations, easy access to data, computing power, human and capital resources, capacity, and budget hinders accurate and timely forecasts—particularly in Latin America and sub-Saharan Africa.

With a global weather cache, forecasters can more easily and freely access local weather observations with a low technical barrier to entry. A partnership with the UK Meteorological Office (UK Met) and the NOAA National Weather Service (NWS) supported by Synoptic Data (Synoptic) and Amazon Web Services (AWS) coalesced from the American Meteorological Society (AMS) Annual Meeting to support a global weather cache built on AWS products and services. AWS Promotional Credit from the Amazon Sustainability Data Initiative (ASDI) helped to support the experimentation and development of a WIS 2.0 global weather cache on AWS.

“Today, one-third of the world’s people, mainly in the least developed countries and small island developing states, are still not covered by early warning systems. This is unacceptable, particularly with climate impacts sure to get even worse. Early warnings and action save lives. To that end, today I announce the United Nations will spearhead new action to ensure every person on Earth is protected by early warning systems within five years,” said UN Secretary-General António Guterres on World Meteorological Day in 2022.

According to the Global Commission on Adaptation, “Early warning systems save lives and assets worth at least ten times their cost. Just 24 hours warning of a coming storm or heat wave can cut the ensuing damage by 30 percent, and spending $800 million on such systems in developing countries would avoid losses of $3–16 billion per year.”

According to the WMO, “Early warning systems are a proven, effective, and feasible climate adaptation measure that save lives, reduce poverty and economic losses, and are shown to provide near tenfold return on investment.”

According to the Manual on WMO Information System Volume II, a WIS 2.0 Global Cache shall provide a highly available storage and download service for accessing discovery metadata records and core data for real-time or near real-time exchange. Additionally, the cache includes a Message Broker, and the WMO core data should be real-time or near real-time and be retained for 24 hours. This WMO core data includes data from WMO members and contributing organizations.

The WIS 2.0 system and the global cache provide a single point of access to improve the speed and accuracy with which forecasts can be generated while decreasing the time and capital requirements. This post describes the value of a global weather cache as well as the design and architecture for building the WIS 2.0 global weather cache on AWS.

Solution overview

The goal of this solution is to build the WIS 2.0 Global Cache from the WIS 2.0 Global Cache system context using AWS services, including serverless technology and cloud-centered solutions following a pub-sub model. The Global Cache provides reliable access to weather data by subscribing to Global Brokers and WIS 2.0 nodes. Data consumers of the WIS 2.0 Global Cache can use a WIS 2.0 subscriber such as the WIS 2.0 in a box, which is a reference implementation of a WMO WIS 2.0 Node. Explore an example WIS 2.0 in a box Synoptic deployment.

Data consumers can also access the data in Open Data on AWS. The AWS Open Data Sponsorship Program makes high-value, cloud-optimized datasets publicly available on AWS. AWS Open Data works with data providers to:

• Democratize access to data by making it available for analysis on AWS.
• Develop new cloud-centered techniques, formats, and tools that lower the cost of working with data.
• Encourage the development of communities that benefit from access to shared datasets.

WIS 2.0 Global Cache users

Globally, weather-related natural disasters have increased fivefold over the past 50 years. More than 91 percent of associated deaths have occurred in developing countries. NWS regional information officers provide decision support to public and private sector leaders in collaboration with authoritative national weather services on climate risk and resilience, specifically the intersection of climate and health, and currently lack access to the core foundational data that these types of analysis rely on. Lack of easy data access prevents them from being able to create early warning systems that are critical to making informed life and death decisions relating to issues such as the health impacts of heat waves—particularly in urban areas—and food insecurity.

With the WIS 2.0 environment, users can log on to the system from any device with internet access and have access to the same standardized, secure, validated, and up-to-date global cache of information that the most advanced meteorological organizations use for their weather models. With this foundational infrastructure, regional information officers can work with other partners to build business models that enable visualization, analytic modeling and simulation capabilities to enable business intelligence that supports better policy and practice related to critical public well-being.

Additional opportunities

The backbone of the WIS 2.0 Global Cache and the WIS 2.0 Node significantly improve the ease and access of global weather data. Once WIS 2.0 is fully and globally accepted, opportunities for advanced and high performance computing (HPC) for improved weather modeling. Also data insights and visualization will greatly improve global access to meaningful weather information that can protect and enhance lives and operations across the planet.

Solution walkthrough: Building the WIS 2.0 global weather cache on AWS

The WIS 2.0 Global Cache solution uses Amazon Simple Queue Service (Amazon SQS), AWS Lambda, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), Amazon Elastic Container Service (Amazon ECS), Amazon Elastic File System (Amazon EFS), Amazon CloudFront, and Elastic Load Balancing (ELB) Network Load Balancer. AWS supplied the UK Met with complimentary Amazon S3 data storage resources through ASDI and Synoptic with AWS services and architecture guidance.

The WIS 2.0 Global Cache is composed of the following components:

• Front-end WIS 2.0 MQTT client subscribes to upstream Global Brokers
• Local Message Broker (MQTT 3.1.1 or MQTT 5.0)
• Global Cache manager, responsible for managing the content and publishing notification messages
• Data storage (with data access API)

Figure 1. Architecture diagram for WIS 2.0 Global Cache including Amazon SQS, AWS Lambda, Amazon DynamoDB, Amazon S3, Amazon ECS, AWS Fargate, Amazon EFS, and ELB Network Load Balancer.

Front-end MQTT client

The front-end WIS 2.0 MQTT client receives messages from the upstream Global Broker and sends the messages into an incoming SQS queue for processing. Amazon ECS with AWS Fargate is used to run a python MQTT in a serverless manner.

Global Cache manager

For the WIS 2.0 Global Cache manager, the following AWS services are used: Amazon SQS for the incoming queue and dead-letter queue; AWS Lambda for deduplication, processing incoming messages, separating data payloads and metadata, writing metadata and identifiers to Amazon DynamoDB, streaming data to Amazon S3 and sending messages to the MQTT Broker on Amazon ECS with AWS Fargate.

Figure 2. WIS 2.0 data viewed in the WIS 2.0 in a box application. The map shows US weather data observation locations in a map along with an observation location’s weather data.

Local Message Broker

The WIS 2.0 Global Cache Local Message Broker is an MQTT bridge and client deployed using Amazon ECS, a fully managed container orchestration service that simplifies the deployment, management, and scaling of containerized applications. The message broker passes the WIS 2.0 message topic structure. Learn more about the details of the WIS 2.0 standard topic structure.

Figure 3. Example screenshot of the WIS 2.0 Global Cache in MQTT Explorer.

Data Storage

In the WIS 2.0 Global Cache solution on AWS, Amazon S3 is being used for raw weather data storage. The Amazon Sustainability Data Initiative (ASDI) is supporting this data storage as part of its commitment to making climate-relevant data easier to access and analyze. Data for the WIS 2.0 Global Cache will be retained for 24 hours and will draw the benefits of scalability, data availability, security, and performance of Amazon S3.

Benefits of the WIS 2.0 Global Cache

“With WIS 2.0 and the new Unified Data Policy, WMO is democratizing authoritative weather, climate, and water data published by its 193 member countries and territories. The WIS 2.0 Global Caches ensure that observation and prediction data from across the globe is highly available and freely accessible to all like never before,” said Jeremy Tandy, a principal fellow at the UK Met Office.

The Met Office and NOAA are jointly hosting a WIS 2.0 Global Cache with support from AWS. AWS provided both engineering support to make the best of their managed services and cloud-native capabilities and are delivering up to 100 gigabytes (GB) per day for free through ASDI. More than just storage and egress, Amazon’s global infrastructure makes sure that essential data gets to those who need it fast. And seconds matter when delivering the safety-critical services that rely on this data. This is a fantastic example of public and private organizations working together to provide the public goods infrastructure that is a foundation of our response to ever-increasing extreme weather and climate change.

Conclusion

This post highlights the importance of a global weather cache and architecture for building a WIS 2.0 global weather cache on AWS. Given the current increasing severity of weather and global climate model projections, it is ever more critical to expose global weather data to better understand and predict weather and severe weather events. With WIS 2.0, it is possible to grow the geographical scope of data contributors and attract resources and capacity to build more weather-related insight from democratized global weather data, thus saving and improving lives and activities. The data served out to the global weather community will enable further insights and groundbreaking advances to make weather prediction timelier and more accurate.