AMS Weekly Seminar | Sayan Banerjee

Location: Shaffer 3

When: April 2nd at 1:30 p.m.

Title: Stationary States and Phase Transitions in Noisy Mean-Field Transformers

Abstract: Mean-field limits of attention-based models—such as noisy Transformer architectures—lead naturally to McKean–Vlasov equations on the circle. Understanding their stationary states is key to explaining phase transitions, symmetry breaking, and the emergence of structured multi-mode patterns in these systems.

We develop a Fourier-based framework for analyzing stationary solutions of such equations. The central observation is an exact equivalence between the stationary McKean–Vlasov equation and an infinite-dimensional quadratic system for its Fourier coefficients. This reformulation converts the problem from a nonlinear PDE in function space to an explicit algebraic system in sequence space, allowing detailed analysis of bifurcations from the uniform state.

We derive analytic criteria describing when and how stationary solutions emerge, including supercritical and subcritical bifurcations involving multiple interacting modes, and connect these mechanisms to discontinuous (first-order) phase transitions. At the global level, we characterize the geometry of the free-energy landscape and identifyphase transitions with non-differentiability of the minimal free energy.

Specializing to the Noisy Mean-Field Transformer model, we show how increasing the inverse temperature parameter β produces a rich family of approximate multi-mode stationary states (interpretable as metastable configurations) and drives a sharp transition from continuous to discontinuous phase behavior.

Based on joint work with Krishnakumar Balasubramanian and Philippe Rigollet.

Zoom link: https://wse.zoom.us/j/92366532431