Self-distillation

What it is

During this process, a model generates inference outputs on a dataset, which are then used as the ground truth for a subsequent round of training. The model essentially teaches itself by distilling its own probability distributions into a cleaner or more compact form. This often involves thousands to millions of inference passes to generate the training data. The primary goal is to capture the model's high-confidence predictions while minimizing noise found in manual labels.

Why it matters

You should care because self-distillation can improve a smaller model's performance without requiring expensive human-annotated data. If you are managing model costs, using self-distillation allows you to deploy smaller, faster, and cheaper models that match the output quality of much larger, more expensive ones. Ignoring this technique might lead you to overspend on high-latency models when a self-distilled compact model could handle the task.

In practice

In practice, you observe this as a fine-tuning phase where the dataset consists entirely of your own model's synthetic outputs. You adjust the temperature setting of your inference calls to ensure the generated training data is diverse yet accurate before feeding it back into the training loop.

The tradeoff

The primary risk is model collapse where the model begins to reinforce its own errors or hallucinations rather than improving its reasoning capabilities.

Where it appears

Research summaries that use Self-distillation, each linked to its source paper.

• Improving AI Text Recognition for Hard Images
  ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

  ArmorOCR improves how AI models read adversarial text in images by using a specialized training process and a new benchmark for evaluating robustness.

• Self-Evolving GUI Visual Grounding Agents
  Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

  The paper introduces a framework that allows GUI visual grounding models to continuously improve after deployment by learning from their own exploration failures through reflection-guided self-distillation.

• Improving Multimodal Models Without External Labels
  Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots

  The researchers developed a self-distillation technique for multimodal large language models that sharpens visual perception by identifying and training on internal counterfactual blind spots.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Improving Multilingual Reasoning in Language Models
  RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

  The paper introduces RP-OPSD, a method that improves how language models transfer English reasoning skills to low-resource languages by selectively applying privileged distillation based on reasoning-pivot signals.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• Efficient Listwise Reranking for Diverse Data
  jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation

  Jina-reranker-v3.5 introduces a hybrid attention architecture and self-distillation protocol to improve retrieval performance across varied and semi-structured domains.

• Optimizing Agent Tool Selection for Mobile
  MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking

  MagicSelector improves agent tool retrieval in mobile environments by using counterfactual reasoning and progressive reranking to eliminate semantic mismatches.

• Improving AI Text Recognition for Hard Images
  ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

  ArmorOCR improves how AI models read adversarial text in images by using a specialized training process and a new benchmark for evaluating robustness.

• Making AI Models Learn Their Own Context
  Latent On-Policy Self-Distillation

  The researchers developed a method to replace hand-designed improvement rules with a system that learns to generate its own contextual guidance for model training.

• Self-Evolving GUI Visual Grounding Agents
  Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

  The paper introduces a framework that allows GUI visual grounding models to continuously improve after deployment by learning from their own exploration failures through reflection-guided self-distillation.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Adaptive Supervision for Better Reasoning Models
  DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

  The researchers developed a method called DASH that dynamically adjusts how reasoning models learn from their own outputs to produce more accurate results.

• Improving Multilingual Reasoning in Language Models
  RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

  The paper introduces RP-OPSD, a method that improves how language models transfer English reasoning skills to low-resource languages by selectively applying privileged distillation based on reasoning-pivot signals.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• Improving Multimodal Models Using Visual Distillation
  OPD-V: Visual On-Policy Self-Distillation with Modality Balance

  OPD-V improves multimodal model performance and reduces latency by using visual-based self-distillation to balance how the model uses image and text data.

• Efficient Listwise Reranking for Diverse Data
  jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation

  Jina-reranker-v3.5 introduces a hybrid attention architecture and self-distillation protocol to improve retrieval performance across varied and semi-structured domains.