Himawari-8: Enabling access to key weather data

Image from Himawari-8; photo courtesy NOAA.

The National Environmental Satellite, Data, and Information Service (NESDIS), a part of the National Oceanic and Atmospheric Administration (NOAA), has the mission to provide secure and timely access to global environmental data and information from satellites and other sources, to promote and protect the United States’ security, environment, economy, and quality of life. NOAA NESDIS is now making Himawari-8, a key foundational weather dataset, available on AWS.

As part of the Amazon Sustainability Data Initiative, we invited Manan Dalal, technology architect and cloud technical director for NESDIS, and Jena Kent, communications lead/National Ocean Service co-technical data lead for NOAA’s Big Data Program (BDP) and oceanographer for NOAA’s Center for Operational Oceanographic Product and Services, to share how NOAA is using Amazon Web Services (AWS) to provide better access to critical weather data.

Enabling better access to foundational weather data

Last December, AWS announced the expansion of its collaboration with the U.S. National Oceanic and Atmospheric Administration (NOAA) to make environmental data easier to access and use through the Big Data Program (BDP). Users can now access new, authoritative NOAA data on AWS without needing to download and store their own copies. Researchers and entrepreneurs can deploy compute resources on-demand in the cloud, perform analysis quickly and efficiently, and save costs by letting researchers ask more questions and experiment more easily.

One of the foundational datasets now available through this collaboration is Himawari-8, the Japan Meteorological Agency’s (JMA’s) satellite dataset. NESDIS is now the single source of the Himawari-8 dataset in the US. The dataset is available on the Registry of Open Data on AWS.

Using the cloud to accomplish the NESDIS mission

As part of a larger cloud implementation, NESDIS designed the NESDIS Common Cloud Framework (NCCF). One of the first services operationalized in the NCCF is the Consolidated Ingest Service, which securely brings in foreign and commercial data. The NCCF Consolidated Ingest Service pulls Himawari-8 data from JMA and inspects the data for viruses and other malicious code. The inspected data is used to produce other NOAA products that are delivered to the user community including the National Weather Service (NWS) and other operational partners. Simultaneously, the data is made available to BDP partners like AWS that make the data more easily accessible to the broader community.

Previously, the traditional method of disseminating foreign and commercial datasets to the public was through on-premises hardware with limitations on network bandwidth and hardware resources. The collaboration with AWS via the BDP provides an efficient, cost-effective, and reliable mechanism for disseminating data to the public. This method allows NESDIS and NOAA to make more datasets available to the public than previously planned since it provides a scalable approach to staging and analyzing data using the cloud. The mechanism to disseminate this data through the BDP has the potential to revolutionize the way NESDIS delivers data to the public by staging data in the cloud next to computing capabilities, which minimizes the need to move data around.

Changing the paradigm of data dissemination

Through the BDP, NESDIS can inspect data like Himawari-8 for security threats before it enters their mission-critical systems. Using the AWS Cloud allows NESDIS to establish a new enterprise capability for inspection of foreign and commercial data in a secure and reliable manner.

The cloud helps NESDIS avoid procuring additional on-premises hardware and can now scale compute usage based on need. The redundant and highly available services in the cloud also leads to robust, fault-tolerant service. This approach enables faster implementation compared to an on-premises solution. Currently, the NCCF v1.0 service is operational and handles about 41,000 files per day from JMA with low latency and high availability.

Additionally, the NCCF is fed from a large redundant network with diverse network providers and AWS Direct Connect, which provides a reliable and highly available network architecture. The NCCF v1.0 services uses AWS to deploy services using agile processes and blue-green deployment techniques. The services are monitored via various dashboards fed from Kibana, an open source visualization dashboard, with alerts and notifications delivered to NESDIS helpdesk and analysts.

Impact: the potential to improve weather forecasts

Making Himawari-8 data readily available to the research community on AWS provides opportunities for forecast applications and facilitates collaboration between research groups. Himawari-8’s Advanced Himawari Imager (AHI) has many new channels (total of 16) compared with previous geostationary satellites. Wider access to this dataset can help meteorologists improve weather analysis and forecasting.

For example, one application of this data is tropical cyclone analysis and forecasting. Himawari-8 imagery provides coverage of the western North Pacific—the most active tropical cyclone basin on the globe—and parts of the southern Hemisphere and Indian Ocean. This makes Himawari-8 data valuable for National Meteorological and Services and tropical cyclone research. GOES-16/17 data is already accessible on AWS and the addition of Himawari-8 will allow applications of this modern generation of satellite data to be used to advance the understanding of concepts such as tropical cyclone intensity change and further drive their integration into the forecasting process.

Learn more about the NOAA’s Big Data Program and the Amazon Sustainability Data Initiative (ASDI).