Optimizing Muon for Low Rank Adapters

Summary & Methodology Analysis

The primary challenge in combining Muon with LoRA is the inability to orthogonalize weight updates within a low rank parameterization. The authors address this by approximating the Muon objective through linearizing the function and dropping second order terms. By solving this linearized objective with least squares, the method determines descent steps for the A and B adapters, effectively bypassing the need for full scale orthogonalization during parameter efficient fine-tuning. This allows for a structured approach to updating weights while maintaining first order dynamics and reparameterization invariance. The algorithm, termed sMuon, relies on a matmul-only computation involving matrix inverse roots and the matrix sign function. This ensures that the process remains computationally feasible for large models by avoiding the high overhead associated with orthogonalizing full sized weight matrices. The team also incorporates split weight decay and momentum projection to refine the update process. Despite the gains in performance, the paper notes that a naive implementation remains computationally inefficient. The authors highlight that the results are sensitive to the chosen base model and the specific evaluation task, indicating that performance is not universal across all applications.