Transferring Reasoning Capabilities Between Language Models
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 7 concepts
Key Takeaways
- SimpleOPD successfully enables distillation between different model families and tokenizers by aligning response tokens in a shared text space.
- Intern-S2-Preview gained 21.2 points on the ProofBench dataset, outperforming the Gemini-2.5-Pro model.
- Adding student reference KL loss significantly improved performance, driving AIME25 scores for Intern-S2-Preview up to 95.80.
- The method demonstrates consistent gains across diverse models, including a 12.30 point increase on ProofBench for Qwen3-4B-OPD.
Summary & Methodology Analysis
SimpleOPD operates by aligning the reasoning paths of a teacher model, like SU-01, with a student model. Because different models use different tokenizers (the mechanism that converts raw text into numerical representations for the model), the method reconstructs the teacher context using its specific chat template and encodes the student response under the teacher tokenizer. It then identifies a partial one-to-one mapping in the shared text space, where both models have produced the same text span. Positions that align inherit the teacher log-probability (the model's confidence in selecting a specific token), while unmatched positions default to the student's own values. The final objective is computed using a PPO clipped policy loss (a technique from reinforcement learning that prevents large, unstable updates to model weights during training) with fixed policy advantages, including a student reference KL loss to prevent the model from drifting too far from its original behavior.
Performance metrics highlight the efficacy of this approach. Intern-S2-Preview reached a score of 55.2 on ProofBench, surpassing Gemini-2.5-Pro. When combining the method with KL regularization, Intern-S2-Preview achieved an AnswerBench score of 79.10 and an AIME25 score of 95.80. The approach is versatile, working across various architectures like Qwen3 and Intern-S2, with the latter showing a 22.80 point improvement over its base model when using the distillation framework.
However, the paper notes several operational limitations. Direct use of this on-policy distillation can cause rapid response-length growth and frequent truncation of reasoning, which risks destabilizing the training process. The authors observe that masking structural termination tokens is insufficient to solve this length expansion problem. Furthermore, there is a clear relationship where larger gaps in reasoning capability between the teacher and student require stronger KL regularization, and tokenizer discrepancies between model families make the transfer of capabilities more technically challenging.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of SimpleOPD?
The goal is to distill long-context mathematical reasoning from a teacher model into smaller student models that use different tokenizers and model architectures.
Q2. Does this method work across different types of AI models?
Yes, the method is designed to be tokenizer-agnostic, allowing it to transfer capabilities across different model families and architectures.
Q3. What kind of performance gains were observed?
Models like Intern-S2-Preview showed significant improvements, such as a 21.2 point gain on ProofBench and reaching a 95.80 score on AIME25.
Q4. How does the model handle differences in how teachers and students process text?
It performs token alignment by identifying a partial one-to-one mapping in the shared text space where both models have produced the same text span.
Q5. What is the role of KL loss in this method?
A student reference KL loss is added to the objective to constrain excessive policy drift during the distillation process.
Q6. What happens if the teacher and student models have very different capabilities?
The paper notes that larger teacher-student gaps generally require stronger KL regularization.
Q7. Are there any risks to training stability with this method?
Yes, direct on-policy distillation can cause rapid response-length growth, leading to frequent truncation and incomplete reasoning, which can destabilize training.
Q8. Does masking special tokens solve the length expansion problem?
No, the paper states that special-token masking alone cannot fully resolve the length expansion problem.
Q9. Does the paper mention the specific hardware used for training?
No, the paper does not specify the hardware requirements.