Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”
Sign in
Sign in
or
Create a new account
About
Categories
Delivery Methods
Solutions
Resources
 Your Saved List Become a Channel Partner Sell in AWS Marketplace Amazon Web Services Home Help

NOAA Cloud Optimized Zarr Reference Files (Kerchunk)

Provided by: NOAA's National Ocean Service, the Integrated Ocean Observing System (IOOS) , part of the Amazon Sustainability Data Initiative

NOAA Cloud Optimized Zarr Reference Files (Kerchunk)

Provided by: NOAA's National Ocean Service, the Integrated Ocean Observing System (IOOS) , part of the Amazon Sustainability Data Initiative

This product is part of the Amazon Sustainability Data Initiative and contains data sets that are publicly available for anyone to access and use. No subscription is required. Unless specifically stated in the applicable data set documentation, data sets available through the Amazon Sustainability Data Initiative are not provided and maintained by AWS.

Description

This repository contains references to datasets published to the NOAA Open Data Dissemination Program. These reference datasets serve as index files to the original data by mapping to the Zarr V2 specification . When multidimensional model output is read through zarr, data can be lazily loaded (i.e. retrieving only the data chunks needed for processing) and data reads can be scaled horizontally to optimize object storage read performance. The process used to optimize the data is called kerchunk . RPS runs the workflow  in their AWS cloud environment every time a new data notification is received from a relevant source data bucket. These are the current datasets being cloud-optimized. Refer to those pages for file naming conventions and other information regarding the specific model implementations: NOAA Operational Forecast System (OFS) 
NOAA Global Real-Time Ocean Forecast System (Global RTOFS) 
NOAA National Water Model Short-Range Forecast  Filenames follow the source dataset’s conventions. For example, if the source file is nos.dbofs.fields.f024.20240527.t00z.nc Then the cloud-optimized filename is the same, with “.zarr” appended nos.dbofs.fields.f024.20240527.t00z.nc.zarr Data Aggregations We also produce virtual aggregations to group an entire forecast model run, and the “best” available forecast. Best Forecast (continuously updated) - nos.dbofs.fields.best.nc.zarr Full Model Run - nos.dbofs.fields.forecast.[YYYYMMDD].t[CC]z.nc.zarr

  • CC is the model run cycles, 00, 06, 12, 18 , or 03, 09, 15, 21 for nowcast and forecast runs
  • YYYY = year, MM = month, DD = day Cloud optimization workflows supported by RPS Group , a Tetra Tech Company
License

NOAA data disseminated through NODD is made available under the [Creative Commons 1.0 Universal Public Domain Dedication (CC0-1.0) license](https://creativecommons.org/publicdomain/zero/1.0/?ref=chooser-v1\ ), which is well-known and internationally recognized. There are no restrictions on the use of the data. The data are open to the public and can be used as desired. NOAA has adopted the Creative Commons license to ensure maximum use of our data, to spur and encourage exploration and innovation throughout the industry. This license is applicable to each of the NOAA datasets made available by NODD. NOAA requests attribution for the use or dissemination of unaltered NOAA data. However, it is not permissible to state or imply endorsement by or affiliation with NOAA. If you modify NOAA data, you may not state or imply that it is original, unaltered NOAA data.

Documentation

Refer to source datasets documentation

How to cite

NOAA Cloud Optimized Zarr Reference Files (Kerchunk) was accessed on DATE from https://registry.opendata.aws/noaa-nodd-kerchunk .

Update frequency
Optimizations run every time new data is uploaded to the source buckets and are available here within minutes.
Support information

Managed by: NOAA's National Ocean Service, the Integrated Ocean Observing System (IOOS) 

Contact: For questions regarding data content or quality, visit Email The Tetra Tech Team. For any questions regarding data delivery or any general questions regarding the NOAA Open Data Dissemination (NODD) Program, email the NODD Team at nodd@noaa.gov. We also seek to identify case studies on how NOAA data is being used and will be featuring those stories in joint publications and in upcoming events. If you are interested in seeing your story highlighted, please share it with the NODD team by emailing nodd@noaa.gov

General AWS Data Exchange support
Contact Us 

Resources on AWS

Description

Cloud-optimized Zarr Reference Files

Resource type
S3 Bucket
Amazon Resource Name (ARN)
arn:aws:s3:::noaa-nodd-kerchunk-pds
AWS Region
us-east-1

AWS CLI Access (No AWS account required)

aws s3 ls --no-sign-request s3://noaa-nodd-kerchunk-pds/
Explore
Description

New data notifications for Cloud-optimized Zarr Reference Files

Resource type
SNS Topic
Amazon Resource Name (ARN)
arn:aws:sns:us-east-1:123901341784:NewNODDKerchunkObject
AWS Region
us-east-1

Usage examples

Links

View this dataset on Github 

Tell us about your project