What limits the ability of scientific machine learning models to predict events never seen during training? I will describe my group’s efforts to understand out-of-distribution generalization in dynamical systems. We first train and analyze small learning models, and obtain scaling laws relating generalization to the ability of models to learn in-context estimates of dynamical operators. We then scale up, and train a time series foundation model on a dataset comprising hundreds of thousands of dynamical systems discovered via evolutionary search. We find that generalization in large models is enabled by a complex set of internal mechanisms, including zero-shot transfer across scales, in-context learning of transfer operators, and a neural scaling law relating performance to the diversity of dynamics encountered during training. The representations of dynamics learned by our model are sufficiently robust to provide informative representations of real-world systems biology time series, which we demonstrate by predicting indirect regulatory logic in large-scale gene expression measurements. Our work shows the potential of large-scale learning models to enable new ways of characterizing and forecasting complex dynamics.
Speaker
William GilpinAssistant Professor of Physics at UT Austin