Increasingly powerful AI models are demonstrating notable accuracy in short-term weather forecasting. However, a new study led by scientists from the University of Chicago, in collaboration with New York University and the University of California Santa Cruz, reveals that these neural networks face limitations when predicting unprecedented weather events. The research, published on May 21 in the Proceedings of the National Academy of Sciences, highlights that AI models cannot forecast beyond their training data's scope, potentially excluding rare events like 200-year floods or massive hurricanes.
This limitation is crucial as researchers integrate neural networks into operational weather forecasting and risk assessments. The authors suggest incorporating more math and physics into AI tools to address this issue. "AI weather models are one of the biggest achievements in AI in science. What we found is that they are remarkable, but not magical," said Pedram Hassanzadeh, an associate professor at UChicago and a corresponding author on the study.
The study explored how AI models handle "gray swan" events—rare but locally devastating occurrences—by testing their ability to predict hurricanes. Researchers trained a neural network with decades of weather data but excluded hurricanes stronger than Category 2. When fed conditions leading to a Category 5 hurricane, the model underestimated its strength. "It always underestimated the event," said Yongqiang Sun, a research scientist at UChicago and co-author.
Traditional weather models incorporate physics principles to understand atmospheric dynamics, unlike neural networks which rely solely on past patterns. No major service currently uses only AI for forecasting; however, as usage expands, addressing these limitations becomes vital.
Researchers propose merging mathematical tools with atmospheric physics principles into AI-based models to improve predictions of extreme events. "The hope is that if AI models can really learn atmospheric dynamics, they will be able to figure out how to forecast gray swans," Hassanzadeh stated.
A promising approach under exploration is active learning—where AI aids traditional physics-based models in generating examples of extreme events for better training data placement. Jonathan Weare from NYU emphasized smarter data generation methods: "We need to think about smarter ways to generate data."
Co-authors include University of Chicago Prof. Dorian Abbot and computational scientist Mohsen Zand, alongside Ashesh Chattopadhyay from UC Santa Cruz. The study utilized resources from the University of Chicago Research Computing Center.
Citation: “Can AI weather models predict out-of-distribution gray swan tropical cyclones?” Sun et al., Proceedings of the National Academy of Sciences, May 21, 2025.
Funding sources include the Office of Naval Research, Army Research Office, and National Science Foundation.