Adaptive Supervision for Better Reasoning Models

Summary & Methodology Analysis

Standard on-policy self-distillation, which is the process of training a smaller model to mimic a larger model using its own generated outputs, often treats all parts of a reasoning rollout equally. This approach fails to account for the reality that some parts of a reasoning process are more difficult or critical than others. DASH addresses this by identifying student-generated prefixes and comparing them to privileged teacher distributions. It calculates the forward KL divergence, a mathematical measure of how one probability distribution differs from another, at each token position and determines the gap between that local signal and the sequence-level average. These gaps are then mapped to adaptive gates that govern how distillation loss is propagated backward through the model, allowing the training process to focus more effectively on segments with the highest discrepancies. This allows the model to refine its internal logic more precisely during the training loop. The methodology relies on a multi-step aggregation of these local signals to create path-dependent supervision weights. While the paper demonstrates consistent gains across benchmarks including AIME 2024, AIME 2025, and HMMT February 2025, it does note specific architectural trade-offs. The fixed-horizon gradient decomposition does not incorporate trajectory score-function terms or future-to-past credit assignment, meaning it may miss opportunities to optimize across the entire sequence lifecycle. Additionally, the observed performance improvements are sensitive to the specific type of distillation divergence chosen, suggesting that the configuration of the loss function remains a critical factor for practitioners deploying this method.