Welcome to Abstracts, a Microsoft Research Podcast that puts the spotlight on world-class research in brief. I’m Amber Tingle. In this series, members of the research community at Microsoft give us a quick snapshot—or a podcast abstract—of their new and noteworthy papers.
[MUSIC FADES]
Our guests today are Megan Stanley and Wessel Bruinsma. They are both senior researchers within the Microsoft Research AI for Science initiative. They are also two of the coauthors on a new Nature publication called “A Foundation Model for the Earth System.” This is such exciting work about environmental forecasting, so we're happy to have the two of you join us today. Megan and Wessel, welcome.
Thank you. Thanks. Great to be here.
Thanks. TINGLE: Let's jump right in. Wessel, share a bit about the problem your research addresses and why this work is so important.
I think we're all very much aware of the revolution that's happening in the space of large language models, which have just become so strong. What's perhaps lesser well-known is that machine learning models have also started to revolutionize this field of weather prediction. Whereas traditional weather prediction models, based on physical laws, used to be the state of the art, these traditional models are now challenged and often even outperformed by AI models.
This advancement is super impressive and really a big deal. Mostly because AI weather forecasting models are computationally much more efficient and can even be more accurate. What's unfortunate though, about this big step forward, is that these developments are mostly limited to the setting of weather forecasting. Weather forecasting is very important, obviously, but there are many other important environmental forecasting problems out there, such as air pollution forecasting
or ocean wave forecasting. We have developed a model, named Aurora, which really kicks the AI revolution in weather forecasting into the next gear by extending these advancements to other environmental forecasting fields, too. With Aurora, we're now able to produce state-of-the-art air pollution forecasts using an AI approach. And that wasn't possible before!
Megan, how does this approach differ from or build on work that's already been done in the atmospheric sciences?
Current approaches have really focused training very specifically on weather forecasting models. And in contrast, with Aurora, what we've attempted to do is train a so-called foundation model for the Earth system. In the first step, we train Aurora on a vast body of Earth system data. This is our pretraining step.
And when I say a vast body of data, I really do mean a lot. And the purpose of this pretraining is to let Aurora, kind of, learn some general-purpose representation of the dynamics that govern the Earth system. But then once we've pretrained Aurora, and this really is the crux of this, the reason why we're doing this project, is after the model has been pretrained, it can leverage this learned general-purpose representation and
efficiently adapt to new tasks, new domains, new variables. And this is called fine-tuning. The idea is that the model really uses the learned representation to perform this adaptation very efficiently, which basically means Aurora is a powerful, flexible model that can relatively cheaply be adapted to any environmental forecasting task.
Wessel, can you tell us about your methodology? How did you all conduct this research?
While approaches so far have trained models on primarily one particular data set, this one dataset is very large, which makes it possible to train very good models. But it does remain only one dataset, and that's not very diverse. In the domain of environmental forecasting, we have really tried to push the limits of scaling to large data by training Aurora on not just this one large dataset, but on as many very large datasets as we could find.
These datasets are a combination of estimates of the historical state of the world, forecasts by other models, climate simulations, and more. We've been able to show that training on not just more data but more diverse data helps the model achieve even better performance. Showing this is difficult because there is just so much data. In addition to scaling to more and more diverse data, we also increased the size
of the model as much as we could. Here we found that bigger models, despite being slower to run, make more efficient use of computational resources. It's cheaper to train a good big model than a good small model. The mantra of this project was to really keep it simple and to scale to simultaneously very large and, more importantly, diverse data and large model size.
So, Megan, what were your major findings? And we know they're major because they're in Nature. [LAUGHS]
Yeah, [LAUGHS] I guess they really are. So the main outcome of this project is we were actually able to train a single foundation model that achieves state-of-the-art performance in four different domains. Air pollution forecasting. For example, predicting particulate matter near the surface or ozone in the atmosphere. Ocean wave forecasting, which is critical for planning shipping routes.
Tropical cyclone track forecasting, so that means being able to predict where a hurricane or a typhoon is expected to go, which is obviously incredibly important, and very high-resolution weather forecasting. And I've, kind of, named these forecasting domains as if they're just items in a list, but in every single one, Aurora really pushed the limits of what is possible with AI models. And we're really proud of that.
But perhaps, kind of, you know, to my mind, the key takeaway here is that the foundation model approach actually works. So what we have shown is it's possible to actually train some kind of general model, a foundation model, and then adapt it to a wide variety of environmental tasks. Now we definitely do not claim that Aurora is some kind of ultimate environmental forecasting model. We are sure that the model and the pretraining procedure can
actually be improved. But, nevertheless, we've shown that this approach works for environmental forecasting. It really holds massive promise, and that's incredibly cool.
Wessel, what do you think will be the real-world impact of this work?
Well, for applications that we mentioned, which are air pollution forecasting, ocean wave forecasting, tropical cyclone track forecasting, and very high-resolution weather forecasting, Aurora could today be deployed in real-time systems to produce near real-time forecasts. And, you know, in fact, it already is. You can view real-time weather forecasts by the high-resolution version of the model on the website of ECMWF (European Centre for Medium-Range Weather Forecasts).
But what's remarkable is that every of these applications took a small team of engineers about four to eight weeks to fully execute. You should compare this to a typical development timeline for more traditional models, which can be on the order of multiple years. Using the pretraining fine-tuning approach that we used for Aurora, we might see significantly accelerated development cycles for environmental forecasting problems. And that's exciting.
Megan, if our listeners only walk away from this conversation with one key talking point, what would you like that to be? What should we remember about this paper?
The biggest takeaway is that the pretraining fine-tuning paradigm, it really works for environmental forecasting, right? So you can train a foundational model, it learns some kind of general-purpose representation of the Earth system dynamics, and this representation boosts performance in a wide variety of forecasting tasks. But we really want to emphasize that Aurora only scratches the surface of what's actually possible.
So there are many more applications to explore than the four we've mentioned. And undoubtedly, the model and pretraining procedure can actually be improved. So we're really excited to see what the next few years will bring.
Wessel, tell us more about those opportunities and unanswered questions. What's next on the research agenda in environmental prediction?
Well, Aurora has two main limitations. The first is that the model produces only deterministic predictions, by which I mean a single predicted value. For variables like temperature, this is mostly fine. But other variables like precipitation, they are inherently some kind of stochastic. For these variables, we really want to assign probabilities to different levels of precipitation rather than predicting only a single value.
An extension of Aurora to allow this sort of prediction would be a great next step. The second limitation is that Aurora depends on a procedure called assimilation. Assimilation attempts to create a starting point for the model from real-world observations, such as from weather stations and satellites. The model then takes the starting point and uses it to make predictions.
Unfortunately, assimilation is super expensive, so it would be great if we could somehow circumvent the need for it. Finally, what we find really important is to make our advancements available to the community. [MUSIC]
Great. Megan and Wessel, thanks for joining us today on the Microsoft Research Podcast.
Thanks for having us.
Yeah, thank you. It's been great.
You can check out the Aurora model on Azure AI Foundry. You can read the entire paper, “A Foundation Model for the Earth System,” at aka.ms/abstracts. And you'll certainly find it on the Nature website, too. Thank you so much for tuning in to Abstracts today. Until next time.
[MUSIC FADES]