Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

Overview & Synopsis

Muon has achieved strong empirical performance in large-scale language-model training, motivating a better theoretical understanding of its practical implementation. Unlike the idealized polar-factor update, practical Muon uses a fixed number of quintic Newton--Schulz iterations. Existing analyses mainly establish average- or best-iterate stationarity guarantees, leaving last-iterate convergence less understood. We study last-iterate gradient convergence of practical finite-step Muon on smooth, lower-bounded nonconvex objectives. For deterministic gradients, we show that without input stabilization, Muon can cycle even on a smooth convex quadratic. With positive stabilization, we establish last-iterate gradient convergence for any fixed momentum coefficient below one and a sufficiently small constant learning rate, together with square-summability of gradients. For stochastic gradients, we first show that persistent bounded, unbiased noise can prevent convergence with fixed learning rates and momentum, even with positive stabilization. We then establish almost-sure and mean-square last-iterate gradient convergence by combining decreasing learning rates with increasing momentum under bounded-variance stochastic gradients. Finally, we construct a smooth counterexample showing that neither convergence theorem guarantees a pointwise polynomial rate. We further establish density-one convergence rates, guaranteeing polynomial gradient decay for all but a vanishing fraction of iterations.