Building cloud-based community knowledge about machine learning to predict and understand extreme weather

The National Center for Atmospheric Research (NCAR) is a federally funded research and development center sponsored by the National Science Foundation. It engages in large-scale Earth system science research projects in collaboration with the broader university community. NCAR hosts visitors from around the world, develops community models including the Community Earth System Model and the Weather Research and Forecasting Model, and maintains supercomputers, observational systems, and aircraft to support further study on the how the planet works.

As part of the Amazon Sustainability Data Initiative, we invited Dr. David John Gagne, machine learning (ML) scientist at NCAR, to share how open data and machine learning on Amazon Web Services (AWS) are impacting the way we predict and understand extreme weather.

Tell us about your background and what you are currently researching?

I have been working at the intersection of machine learning and atmospheric science since 2007 with a focus on using machine learning to predict and understand extreme weather, such as hail, tornadoes, and hurricanes. I completed my PhD in meteorology at the University of Oklahoma in 2016 and have been at NCAR since then. I am now head of the analytics and integrative machine learning group in the Computational and Information Systems Lab. We are engaged in cross-disciplinary collaborations across NCAR to develop machine learning systems for predicting and explaining severe weather along with multiple projects to emulate computationally expensive earth system process models.

What is the Artificial Intelligence 4 Earth System Science (AI4ESS) Summer School and why is NCAR hosting it?

The AI4ESS Summer School is a week-long virtual educational event aimed at educating the earth system science community about the fundamentals of artificial intelligence (AI), ML, and deep learning; current applications of AI in Earth System Science (ESS), and emerging ML methods that could be broadly useful to the community. This year’s event was hosted virtually between June 22 and 26, and we had an average of 1,500 people per day tune in to watch the lectures from 40 different countries. We recorded participants from North and South America, Africa, Europe, and Asia. The program featured 15 livestreamed lectures from experts at the intersection of AI and ESS.

We also held a week-long hackathon with five challenge problems: nowcasting lightning with GOES-16 satellite data (to help predict storms earlier, enabling more time for people to seek shelter); predicting El Nino Southern Oscillation (ENSO) (to improve predictions of precipitation patterns and downstream effects); emulating the GECKO-A atmospheric chemistry model (to help with more accurate projections of air quality); speeding up the processing of a holographic cloud particle imager (to help with faster processing of data in the study of how well clouds can reflect light and produce precipitation); and emulating warm rain processes in a climate model (to use less compute power in the study the impact of warm rain on the climate). By speeding up computationally intensive algorithms with emulation and enabling easy access to heterogeneous computing, such as GPUs, AI can help promote more sustainable computing in ESS. Over 150 people participated in the hackathon and wrote 33,000 Slack messages. About half of our participants were graduate students and postdocs with the rest spanning from pre-college students to experienced research scientists.

How did you use AWS during the summer school? Did AWS enable you to operate at a new scale?

We received AWS Promotional Credit that enabled us to set up a Jupyterhub cluster managed by Kubernetes for the hackathon with guidance from the Zero to Jupyterhub site. I had previously used Jupyterhub to support one-day machine learning short courses for 50 people but had never scaled a hackathon to over 150 people in different countries for five days until this event. The AWS Promotional Credit allowed me to provide each participant with their own virtual machine with multiple CPUs, a GPU, and sufficient RAM to perform intense analysis and training of machine learning and deep learning models. I was impressed with how easily I was able to scale the cluster to hundreds of nodes in a matter of minutes. I was also impressed with how well it could recover from the occasional computer failure without losing a participant’s data. We also hosted the hackathon datasets on Amazon Simple Storage Service (Amazon S3) and used cloud-optimized data formats like Zarr and Apache Parquet to make data loading faster and more accessible. Hundreds of people accessing the same datasets at the same time proved not to be an obstacle for us.

If someone missed out on the summer school, is there a way for them to still benefit from it?

We posted the slides, lecture recordings, and links to the code on the A14ESS website and on the hackathon GitHub page. We also plan to incorporate the hackathon datasets into future tutorials, short courses, and summer schools.

How about for yourself, what are your future plans for continuing to do machine learning?

For future directions, I am interested in developing explainable AI systems, quantifying uncertainty in AI predictions, and improving visualizations of AI predictions for weather forecasters and researchers. I look forward to working with my team and collaborators to develop ML systems for a wider array of ESS applications.

To learn more about the Amazon Sustainability Data Initiative, visit our webpage. And check out an interview with Dr. David John Gagne on episode #25 of the Fix This podcast.

About David John Gagne

David John Gagne is a machine learning scientist at the National Center for Atmospheric Research (NCAR) in Boulder, Colorado. His research focuses on developing machine learning systems to improve the prediction and understanding of high impact weather and to enhance weather and climate models. He received his PhD in meteorology from the University of Oklahoma in 2016 and subsequently completed an Advanced Study Program postdoctoral fellowship at NCAR in 2018. He has collaborated with interdisciplinary teams to produce machine learning systems for hail, tornadoes, hurricanes, and renewable energy. In order to educate atmospheric science students and scientists about machine learning, he has led a series of interactive short courses and hackathons.