Improving Model Training with Trust Regions

Summary & Methodology Analysis

The research addresses instability in on-policy distillation (a process where a student model learns from a teacher by generating its own data). Early in the training process, students often produce low-quality prefixes that lead teachers to provide guidance on unhelpful or poor data. To fix this, the authors implement a behavior policy defined by a student-centered KL trust region. This constraint ensures that the behavior policy stays within an allowed local deviation from the student, preventing the training process from straying too far into suboptimal state spaces. The researchers compute the optimal mix between teacher and student policies using binary search on the interval from zero to one. This mixture is then applied with an annealed KL budget that decreases linearly to zero over a fixed warmup horizon. The final update uses the teacher-guided prefixes sampled from this optimized behavior policy. The experimental results demonstrate that this approach achieves the highest average pass@1 score across math-reasoning distillation settings when compared to vanilla on-policy distillation and several baselines. The study evaluated these methods across models including Qwen3-1.7B-Base, Qwen3-8B, Qwen3-0.6B-Base, and Qwen3-4B, using datasets such as MATH500, AIME24, AIME25, AMC, Olympiad, GSM8K, and OpenThoughts3-1.2M. Despite the performance gains, the method has two primary limitations. First, the effectiveness has only been verified for specific math-reasoning tasks, and the warmup schedules may not generalize to other domains. Second, the requirement for online teacher decoding during the warmup phase creates an increase in training-time costs.