Improving Model Reasoning With On Policy Delta Distillation

Summary & Methodology Analysis

The paper addresses the underexplored fundamental design of on policy distillation, which is a technique for transferring knowledge from a larger teacher model to a smaller student model, aiming to improve its loss function and learning signal without exposure bias or noisy reward designs. Standard methods generate responses using the student network, score sampled tokens using the teacher, and train the student via KL divergence or token level log probability differences. The novel contribution, On Policy Delta Distillation, defines a new distillation reward called the delta signal. This signal measures the difference between the reasoning tuned teacher and its pre instruction tuning base model. To handle reward directions and form zero centered advantage values, it applies a centering operation that subtracts expected rewards under the policy model sampling probabilities from the delta signal. Furthermore, it applies a joint condition or agreement condition as a stop condition that turns off the delta advantage when it conflicts with the original distillation signal, preventing over training and convergence instability.