Self-distillation

What it is

During this process, a model generates inference outputs on a dataset, which are then used as the ground truth for a subsequent round of training. The model essentially teaches itself by distilling its own probability distributions into a cleaner or more compact form. This often involves thousands to millions of inference passes to generate the training data. The primary goal is to capture the model's high-confidence predictions while minimizing noise found in manual labels.

Why it matters

You should care because self-distillation can improve a smaller model's performance without requiring expensive human-annotated data. If you are managing model costs, using self-distillation allows you to deploy smaller, faster, and cheaper models that match the output quality of much larger, more expensive ones. Ignoring this technique might lead you to overspend on high-latency models when a self-distilled compact model could handle the task.

In practice

In practice, you observe this as a fine-tuning phase where the dataset consists entirely of your own model's synthetic outputs. You adjust the temperature setting of your inference calls to ensure the generated training data is diverse yet accurate before feeding it back into the training loop.

The tradeoff

The primary risk is model collapse where the model begins to reinforce its own errors or hallucinations rather than improving its reasoning capabilities.

Where it appears

Research summaries that use Self-distillation, each linked to its source paper.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Improving Multilingual Reasoning in Language Models
  RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

  The paper introduces RP-OPSD, a method that improves how language models transfer English reasoning skills to low-resource languages by selectively applying privileged distillation based on reasoning-pivot signals.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• How Voice and Typing Errors Impact LLM Performance
  Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations

  The study quantifies how common input errors from keyboards and voice transcription pipelines degrade the task accuracy of LLM agents.

• Improving LLM Agent Reliability Through Self-Distillation
  PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

  The paper introduces PCSD, a training method that improves LLM agent performance by filtering noisy feedback during the reinforcement learning process.