On-policy distillation

What it is

In this process, you first generate a high volume of responses using a lightweight student model rather than a massive teacher model. You then prompt the teacher model to evaluate or correct these student outputs, creating a dataset of improved responses. The student is then fine-tuned on this high-quality, self-generated data. This approach is often used to compress model capabilities, sometimes reducing inference costs by 10x while maintaining performance levels close to the original larger model.

Why it matters

As an engineer, this matters because it allows you to lower the latency and infrastructure costs of your production AI features without sacrificing accuracy. If you rely solely on massive models for every request, your margins will likely suffer at scale. Understanding this technique helps you decide when to invest in training a specialized, smaller model for specific tasks instead of calling an expensive public API for every interaction.

In practice

You typically see this implemented during the fine-tuning pipeline where the training data is dynamic rather than static. Instead of using a fixed dataset, your CI/CD or training job includes a generation step that calls the teacher model to synthesize labels for student-generated inputs. You will observe this in practice as a requirement for custom training infrastructure that handles both inference-heavy generation phases and supervised fine-tuning cycles.

The tradeoff

The primary tradeoff is the risk of model collapse, where the student inherits the teacher's flaws or fails to generalize if the self-generated data lacks sufficient diversity.

Where it appears

Research summaries that use On-policy distillation, each linked to its source paper.

• Securing AI Agents Using On-Policy Distillation
  SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

  SecOPD improves AI agent security against adaptive prompt injection by using on-policy distillation to provide fine-grained training signals that distinguish between trusted instructions and malicious data.

• Transferring Reasoning Capabilities Between Language Models
  SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

  The paper introduces a method called SimpleOPD that distills advanced mathematical reasoning from a high-performance teacher model into smaller student models across different architectures.

• Improving Model Distillation via Task Sensitivity
  CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

  The paper introduces a method called CROP to selectively focus model distillation on task-relevant information by measuring sensitivity to counterfactual prompts.

• Improving Multilingual Translation Without Reference Data
  Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

  The researchers developed a reference-free post-training method to optimize machine translation models using only source-side text, bypassing the need for high-quality parallel data.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Improving LLM Training with Distilled Reinforcement Learning
  Distilled Reinforcement Learning for LLM Post-training

  The paper introduces a refined reinforcement learning method for LLM post-training that improves performance on math benchmarks by dynamically balancing teacher guidance and model autonomy.

• Recovering Pruned LLM Generation Capabilities
  ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  ShortOPD uses a dynamic distillation strategy to fix structural collapse in pruned LLMs by adjusting training rollouts based on model output quality.

• Improving Model Training with Trust Regions
  Trust-Region Behavior Blending for On-Policy Distillation

  The paper introduces a method called Trust Region Behavior Blending to stabilize on-policy distillation by controlling how teachers supervise student models during early training.

• Improving AI Reasoning Through Verifiable Distillation
  On-policy Distillation with Verifiable Reward

  The paper introduces a method called OPDVR that aligns reinforcement learning signal with task success during model distillation to improve reasoning performance.

• Securing AI Agents Using On-Policy Distillation
  SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

  SecOPD improves AI agent security against adaptive prompt injection by using on-policy distillation to provide fine-grained training signals that distinguish between trusted instructions and malicious data.

• Transferring Reasoning Capabilities Between Language Models
  SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

  The paper introduces a method called SimpleOPD that distills advanced mathematical reasoning from a high-performance teacher model into smaller student models across different architectures.

• Improving AI Vision Without External Data
  Self-Supervised Visual On-Policy Distillation

  The paper introduces a self-supervised method to improve vision model performance by distilling knowledge from a student to an EMA teacher without needing ground-truth labels or extra rewards.

• Improving Model Distillation via Task Sensitivity
  CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

  The paper introduces a method called CROP to selectively focus model distillation on task-relevant information by measuring sensitivity to counterfactual prompts.

• Fixing On-Policy Distillation Mismatch
  Mismatch Matters: On-Policy Distillation Beyond Token Agreement

  The paper introduces TIDE, a method to fix model distillation failures caused by degenerate token agreement and teacher-student mismatch.

• Improving Agent Decision Making Through Self-Evolution
  SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

  The paper introduces a method called SEED that improves agentic performance by having the model analyze its own past trajectories to generate dense feedback for training.

• Recovering Pruned LLM Generation Capabilities
  ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  ShortOPD uses a dynamic distillation strategy to fix structural collapse in pruned LLMs by adjusting training rollouts based on model output quality.

• Improving Model Training with Trust Regions
  Trust-Region Behavior Blending for On-Policy Distillation

  The paper introduces a method called Trust Region Behavior Blending to stabilize on-policy distillation by controlling how teachers supervise student models during early training.