Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics

Overview & Synopsis

Instance-dependent early stopping (IES) reduces training computation by skipping backpropagation for examples whose losses have stabilized, but the second-order loss-difference criterion used by IES lacks a theoretical justification. We develop Theory-Guided Instance-Dependent Early Stopping (TG-IES), which analyzes this criterion under explicit assumptions and derives a noise-calibrated mastery threshold and a conditional stability bound for instance exclusion. We show that second-order differencing has a favorable bias-variance trade-off over first- and third-order differences in a specific signal-to-noise regime, and that its signal contains both gradient-drift and curvature terms. TG-IES derives its stopping threshold and patience from training-run statistics rather than using the manually selected threshold of IES. Across CIFAR-10 and CIFAR-100, TG-IES saves 53-58% of backpropagation instances in its main convolutional settings while keeping accuracy within 0.1-0.9 percentage points of full training; in the primary ResNet-18/CIFAR-10 setting, this corresponds to 35.7% FLOP-normalized savings. The method also transfers to ViT-Tiny, where it achieves smaller savings and incurs a modest accuracy cost. These results provide a theoretical basis for second-order instance-dependent early stopping and show that theory-guided calibration can substantially reduce training computation under the assumptions and experimental settings studied.