Unifying Neural Network Training Dynamics

Summary & Methodology Analysis

The paper introduces the Neural Quadratic Form (NQF) to resolve the conflict between abrupt training transitions and smooth scaling laws. By identifying permutation symmetry as a key invariant, the researchers perform a Taylor expansion of the architecture at the initialization point. They define the NQF as the sum of a constant term and the trace of the product of the weights and a structure matrix. This construction maps the complex training process onto the evolution of a single order parameter, allowing for a systematic analysis of learning dynamics. The researchers model this evolution using a generalized Lotka Volterra equation, which holds true when data matrices satisfy specific commutation or isotropic conditions. This framework effectively tracks the training of MLPs, CNNs, and attention heads. The derivation of power law scaling exponents emerges from the way individual modes are activated, linking discrete saddle to saddle transitions to observed aggregate trends. Despite its utility, the NQF model has strict boundaries. Its approximation is only valid for small weight initialization scales. Furthermore, the reliance on three times differentiability means the model may not apply to architectures that utilize rectified activations at the origin or specific normalization layers. Finally, the paper notes that the NQF does not guarantee that the eigenspectrum of the structure matrix will naturally follow a power law tail.