Dresden 2026 – scientific programme
Parts | Days | Selection | Search | Updates | Downloads | Help
DY: Fachverband Dynamik und Statistische Physik
DY 58: Focus Session: Physics of AI – Part II (joint session SOE/DY)
DY 58.5: Talk
Friday, March 13, 2026, 10:45–11:00, GÖR/0226
Dynamics of neural scaling laws in random feature regression — •Jakob Kramp1,2, Javed Lindner1,2, and Moritz Helias1,2 — 1Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Centre, Jülich, Germany — 2Department of Physics, RWTH Aachen University, Aachen, Germany
Training large neural networks reveals signs of universality that hold across architectures. This holds also for overparameterized networks which converge to effective descriptions in terms of Gaussian process regression. Those simplified models, already show one ingredient of universality in form of neural scaling laws. An important ingredient are power-law distributed principal component spectra of the training data.
Past work has therefore studied the dynamics of deterministic gradient flow in linear regression with and without consideration of power-law distributed spectra. Previously, dynamics of gradient flow with power law data in a type of linear random feature model were able to mimic effects of feature learning. Our work differs from the former by presenting an approach that holds for Bayesian inference on Gaussian processes obtained by stochastic Langevin training as well as for deterministic gradient flow with or without regularization by weight decay. We obtain interpretability from an effective mean-field theory that requires fewer order parameters than previous works.
Keywords: Statistical Field Theory; Neural Network; Feature Regression; Bayesian Learning; Scaling Laws