Improving Multilingual Math Reasoning via Distillation

Summary & Methodology Analysis

The researchers investigate On-Policy Distillation (OPD), a technique where a student model mimics a teacher model by aligning its generation probabilities to match the teacher's trajectory, to enhance multilingual math reasoning. The core contribution, On-Policy Delta Distillation (OPD2), refines this process by introducing a delta signal. This signal is computed as the difference between the teacher's logit probabilities (the raw output values before normalization) and the base model's logit probabilities. To prevent convergence issues common with this delta signal, the authors incorporate reward centering to create an advantage and use the original OPD signal as a conditioning criterion. This refinement ensures that the model effectively learns from the teacher's refined reasoning patterns during the distillation process.

The experimental evaluation focuses on models including Qwen3-1.7B, Qwen3-8B, and larger variants like Qwen3-30B-A3B-2507 across datasets such as PolyMath, Global-MGSM, HRM8K, and KSM. Results demonstrate clear improvements in multilingual benchmarks. For instance, in the Qwen3-1.7B model, OPD2 yields consistent gains, including a 4.0 point improvement in Japanese math reasoning tasks. Larger models like Qwen3-8B also see performance boosts, with 3.3 and 3.1 point gains respectively in Korean and Japanese tasks compared to the standard OPD baseline. These improvements suggest that distilling reasoning patterns is effective for scaling logic-heavy tasks across languages.

Despite these performance gains, the paper highlights significant limitations in deployment. A major issue is language drift: while reasoning logic transfers from English-only training, the model often fails to respond in the target language, opting for English instead. Furthermore, for models using thinking-mode features, intermediate reasoning traces remain locked in English. The consistency of these gains also varies, as the reduction in performance gaps between English and Korean is not uniform across all benchmarks. The paper does not provide specific data on training latency or compute-cost metrics, though it focuses primarily on the architecture of the distillation signal.