On-policy distillation

What it is

In this process, you first generate a high volume of responses using a lightweight student model rather than a massive teacher model. You then prompt the teacher model to evaluate or correct these student outputs, creating a dataset of improved responses. The student is then fine-tuned on this high-quality, self-generated data. This approach is often used to compress model capabilities, sometimes reducing inference costs by 10x while maintaining performance levels close to the original larger model.

Why it matters

As an engineer, this matters because it allows you to lower the latency and infrastructure costs of your production AI features without sacrificing accuracy. If you rely solely on massive models for every request, your margins will likely suffer at scale. Understanding this technique helps you decide when to invest in training a specialized, smaller model for specific tasks instead of calling an expensive public API for every interaction.

In practice

You typically see this implemented during the fine-tuning pipeline where the training data is dynamic rather than static. Instead of using a fixed dataset, your CI/CD or training job includes a generation step that calls the teacher model to synthesize labels for student-generated inputs. You will observe this in practice as a requirement for custom training infrastructure that handles both inference-heavy generation phases and supervised fine-tuning cycles.

The tradeoff

The primary tradeoff is the risk of model collapse, where the student inherits the teacher's flaws or fails to generalize if the self-generated data lacks sufficient diversity.

Where it appears

Research summaries that use On-policy distillation, each linked to its source paper.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Smarter Scheduling for Faster Model Training
  Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

  The researchers created a new, adaptive system that speeds up artificial intelligence training by intelligently deciding when to increase the amount of data processed during each step of learning.

• Improving AI Model Training via Behavior Blending
  Trust-Region Behavior Blending for On-Policy Distillation

  The paper introduces a warmup method for on-policy distillation that regulates student behavior during early training to prevent the model from learning on low-quality data.