New AWS Public Data Set – Real-Time and Archived NEXRAD Weather Data

My colleague Ariel Gold wrote the guest post below to introduce the newest AWS Public Data Set.

— Jeff;

You can now access real-time and archival NEXRAD weather radar data as an AWS Public Data Set.

The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in approximately 5 minute intervals from each site. NEXRAD enables severe storm prediction and is used by researchers and commercial enterprises to study and address the impact of weather across multiple sectors. As part of our research agreement with the US National Oceanic and Atmospheric Administration (NOAA), we are making NEXRAD data freely available on Amazon S3.

The real-time feed and full historical archive of NEXRAD Level II data, from June 1991 to present, is now available for anyone to use. Level II is original resolution, base data from the NEXRAD system.

This is the first time the full NEXRAD Level II archive has been accessible to the public on demand. A wide range of customers have expressed interest in this data, including insurance providers, climate researchers, logistics companies, and weather companies. We’re excited to see what you do with it!

You can learn more about the data and how to access it on our NEXRAD on AWS page.

How We’re Sharing the Data
We’ve been testing out new ways to make NEXRAD data easy to use in the cloud. Before I get into some of the details of our approach, here are a couple of radar data terms for the uninitiated. First, “volume scan” refers to the data collected by the Doppler radar site as it scans the atmosphere. The NEXRAD site breaks these volume scans into “chunks” – small packages of data that are quickly transmitted as a real-time feed. The NEXRAD network generates about 1,200 chunks per hour.

We are storing the real-time chunks and archive (volume scan files) Level II data in two public Amazon S3 buckets. Data flows into the chunks bucket via Unidata’s Local Data Manager (LDM) system with minimal latency from the NEXRAD sites. The chunks are then assembled into volume scan files and added to the archive bucket within seconds or minutes of production. This creates a continuously updated, near-real-time archive of volume scan files.

You can find information on the data structure on our NEXRAD on AWS page. You’ll see that the real-time data is hosted in the “unidata-nexrad-level2-chunks” Amazon S3 bucket. Unidata provides data services, tools, and cyberinfrastructure leadership for the earth science community and they have been fantastic collaborators on this project. You can read more about their experience setting up the NEXRAD real-time feed on AWS on their blog.

Getting Started with NEXRAD on AWS
Unidata, The Climate Corporation, and CartoDB have contributed tutorials to help you get started using NEXRAD on AWS. For example, this tutorial from The Climate Corporation shows you how to read and display the NEXRAD Level II archive data from your Python programs.

Unidata has also made the NEXRAD Level II archive data available via their THREDDS Data Server. You can also browse the archive contents via the AWS JavaScript S3 Explorer:

Learn more about ways to use the data on our NEXRAD on AWS page.

Thank You
We’d like to thank our collaborators at NOAA, CICS-NC, Unidata, and The Weather Company who helped us launch this public data set and continue to help make it available. Many others helped test and contribute tools to the data set and we welcome additional contributions. Tell us how you’re using the data via the NEXRAD on AWS page and sign up for updates on the NOAA Big Data Project here.

— Ariel Gold, Program Manager, AWS Open Data