Distillation

What it is

During training, the smaller student model processes the same input data as the teacher model. Instead of just learning from raw labels, the student optimizes its parameters to match the soft probabilities or output tokens produced by the teacher. This transfers the reasoning patterns of massive models into architectures often 10 to 100 times smaller. The resulting model maintains a significant portion of the teacher's capability while requiring substantially less compute and VRAM to serve.

Why it matters

Knowing about distillation changes your deployment strategy from blindly using the most powerful API to selecting a cost-effective model that meets your specific latency requirements. If you ignore it, you likely overspend on infrastructure for tasks that do not require the reasoning capabilities of a frontier model. It allows you to move expensive logic from the edge or server-side bottlenecks into highly responsive local components.

In practice

You typically access distilled models as specific model weights in a library like Hugging Face or as dedicated smaller endpoints provided by model APIs. You observe the benefit in your metrics by seeing reduced time-to-first-token and lower cost per 1k requests while maintaining similar accuracy on your specific task benchmarks. You might select a model with a suffix like -distill or -small to swap out a larger, slower counterpart in your pipeline.

The tradeoff

Distillation effectively compresses knowledge but cannot impart original reasoning capabilities that were not already present in the teacher model's outputs. Practitioners often overestimate how much the student can generalize to data distributions outside of the teacher's original training scope.

Where it appears

Research summaries that use Distillation, each linked to its source paper.

• Efficient Mixture of Experts Text Embeddings
  Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

  The authors introduce a family of sparse embedding models that leverage Mixture-of-Experts architectures and distillation to achieve high throughput and competitive retrieval performance.

• Improving AI Reasoning Through Verifiable Distillation
  On-policy Distillation with Verifiable Reward

  The paper introduces a method called OPDVR that aligns reinforcement learning signal with task success during model distillation to improve reasoning performance.

• WeChat Multi-modal Embedding Technical Report
  WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

  The paper introduces WeMM-Embedding, a series of multimodal models based on Qwen3.5 that achieve state-of-the-art performance on retrieval benchmarks and demonstrate consistent gains in production applications.

• Local Distillation for Interpretable Machine Learning
  Interpretable AI with Local Distillation

  The paper introduces local distillation, a method that improves the prediction accuracy of simple, interpretable linear models by selectively leveraging predictions from complex black-box models.

• Reusable Execution Experience for AI Agents
  LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks

  Researchers introduced a system called EvoMap that distills successful, verifier-confirmed AI task trajectories into reusable Genes to improve performance and reduce token consumption across various model families.

• Efficient Reference Caching for Diffusion Transformers
  Anchoring Instruction Outside Mask: Exact Reference Caching for Efficient In-Context Diffusion Transformers

  The researchers developed a text anchor method to enable high-speed reference image caching in diffusion transformers without sacrificing model performance.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Structured Evidence for Accurate Image Retrieval
  EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

  EviRank improves image retrieval accuracy by replacing unstructured reasoning with a structured, criteria-based verification framework.

• Efficient Unified Image Generation Models
  Swift-Image: Exploring the Performance Frontier of Compact Unified Image Generation Models

  Swift-Image is a compact, unified model designed to handle text-to-image generation and image editing tasks efficiently under strict computational budgets.

• Learning Full 3D Objects from LiDAR
  GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structure

  GhostPoint improves 3D object detection by training models to predict the hidden structure of objects that are partially occluded in LiDAR sensor data.

• Transferring Reasoning Capabilities Between Language Models
  SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

  The paper introduces a method called SimpleOPD that distills advanced mathematical reasoning from a high-performance teacher model into smaller student models across different architectures.

• Improving AI Vision Without External Data
  Self-Supervised Visual On-Policy Distillation

  The paper introduces a self-supervised method to improve vision model performance by distilling knowledge from a student to an EMA teacher without needing ground-truth labels or extra rewards.

• Making AI Models Learn Their Own Context
  Latent On-Policy Self-Distillation

  The researchers developed a method to replace hand-designed improvement rules with a system that learns to generate its own contextual guidance for model training.

• Improving Autoregressive Video Generation Consistency
  Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation

  The paper introduces a distillation method that aligns teacher supervision with causal inference to resolve context mismatches in video generation models.

• Real-Time Streaming Human Animation System
  LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time

  LiveAnimate is a diffusion-based framework that enables stable long-form human animation for real-time applications by optimizing architecture and inference processes.

• Automated Taxomony Expansion Using LLMs
  ReLTEx: Reliable LLM-based Taxonomy Expansion

  ReLTEx improves automated taxonomy expansion by using LLMs for candidate generation combined with a structure-aware classifier to ensure hierarchical consistency.

• Building Omni-modal Models with Native Video
  Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

  Ex-Omni-2D generates multimodal dialogue responses that natively combine text, personalized speech, and reference-conditioned video to overcome the limitations of visually disembodied models.

• Building Smaller Faster Document Retrieval Models
  DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation

  DistilVDR creates compact single-vector document retrieval systems by distilling knowledge from large vision-language models into significantly smaller student encoders.

• Improving Video Generation Speed and Diversity
  DUET: A Diversity-Quality Duet of Distillation Experts for Two-Step Video Generation

  DUET improves video generation quality and diversity in a two-step process by combining two distinct distillation expert models.

• Fixing On-Policy Distillation Mismatch
  Mismatch Matters: On-Policy Distillation Beyond Token Agreement

  The paper introduces TIDE, a method to fix model distillation failures caused by degenerate token agreement and teacher-student mismatch.

• Nesting Language Models for Efficiency
  Matryoshka Language Model Suites

  The paper introduces a framework for nesting smaller sub-models within a larger architecture to reduce training compute and improve speculative decoding performance.

• Exposing Hidden Reasoning Traces in LLMs
  Stealing Reasoning Traces from Proprietary LLM APIs

  Researchers discovered an architectural flaw in how major LLM providers handle encrypted reasoning traces, allowing them to decrypt and expose proprietary data.

• Improving Agent Feedback with Trajectory Distillation
  Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

  The paper introduces a method to improve agent performance by dynamically allocating hindsight feedback across individual decision steps in a multi-turn task.

• Improving Mobile GUI Agents with Hindsight
  The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

  The researchers introduce Gated Hindsight Distillation to help GUI agents learn from future screenshots when current screen data is insufficient for decision making.

• Optimizing Character Animation for Real-Time Applications
  Wan-Animate-2: Pushing the Application Boundaries of Character Animation

  Wan-Animate-2 introduces a new architecture to solve inefficiencies in character animation by decoupling reference streams and enabling more efficient training.

• Incentivizing LLM Reasoning via Pure Reinforcement Learning
  DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

  The paper introduces DeepSeek-R1, demonstrating that large language models can develop advanced reasoning capabilities through pure reinforcement learning without requiring prior supervised fine-tuning.

• Distilling Knowledge Into Smaller Models
  Distilling the Knowledge in a Neural Network

  The paper introduces knowledge distillation, a technique to transfer generalization ability from cumbersome ensembles or large neural networks into smaller deployment-ready models.

• Coach Models Improve Language Model Training
  LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

  The paper introduces an LLM-as-a-Coach method that replaces traditional scalar rewards in reinforcement learning with rich experiential knowledge to train open-ended task models.

• Improving Model Reasoning With On Policy Delta Distillation
  On-Policy Delta Distillation

  The paper introduces On Policy Delta Distillation, a new method that improves how reasoning capabilities are transferred from a teacher model to a student model.

• One-Step Video Object Removal Without Drafts
  From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting

  The researchers developed a model that performs video object removal in a single inference step by eliminating the need for external draft priors.

• Improving Agent Decision Making Through Self-Evolution
  SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

  The paper introduces a method called SEED that improves agentic performance by having the model analyze its own past trajectories to generate dense feedback for training.

• Converting Document Images to Markdown Efficiently
  OvisOCR2 Technical Report

  OvisOCR2 is a model designed to parse visually rich documents into structured Markdown in a single pass.

• Securing AI Agents Using On-Policy Distillation
  SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

  SecOPD improves AI agent security against adaptive prompt injection by using on-policy distillation to provide fine-grained training signals that distinguish between trusted instructions and malicious data.

• Improving AI Text Recognition for Hard Images
  ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

  ArmorOCR improves how AI models read adversarial text in images by using a specialized training process and a new benchmark for evaluating robustness.

• Improving Model Distillation via Task Sensitivity
  CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

  The paper introduces a method called CROP to selectively focus model distillation on task-relevant information by measuring sensitivity to counterfactual prompts.

• Real Time Speech Enhancement With Mamba
  RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

  The paper introduces RT-SEMamba, a fully causal, Mamba-based speech enhancement architecture operating online with a 25 ms algorithmic latency constraint via progressive knowledge distillation.

• Self-Evolving GUI Visual Grounding Agents
  Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

  The paper introduces a framework that allows GUI visual grounding models to continuously improve after deployment by learning from their own exploration failures through reflection-guided self-distillation.

• Improving Multimodal Models Without External Labels
  Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots

  The researchers developed a self-distillation technique for multimodal large language models that sharpens visual perception by identifying and training on internal counterfactual blind spots.

• Distilling Hierarchical Memory for Agent Models
  Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

  The researchers developed a training-free framework that distills complex strategies and tool-calling logic from large teacher models into smaller, more efficient student models.

• Improving Multi-turn AI Agent Training Guidance
  When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

  The paper introduces a routing method for multi-turn AI agents that selectively applies reference guidance only when the agent's current state aligns with known valid task paths.

• Adaptive Supervision for Better Reasoning Models
  DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

  The researchers developed a method called DASH that dynamically adjusts how reasoning models learn from their own outputs to produce more accurate results.

• Improving Multilingual Reasoning in Language Models
  RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

  The paper introduces RP-OPSD, a method that improves how language models transfer English reasoning skills to low-resource languages by selectively applying privileged distillation based on reasoning-pivot signals.

• Improving Agent Decision Making via Recursive Distillation
  AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

  AgentOPSD introduces a recursive self-distillation method to provide granular credit assignment for multi-turn agentic tasks by analyzing turn-level evidence.

• Improving Multimodal Models Using Visual Distillation
  OPD-V: Visual On-Policy Self-Distillation with Modality Balance

  OPD-V improves multimodal model performance and reduces latency by using visual-based self-distillation to balance how the model uses image and text data.

• Efficient Listwise Reranking for Diverse Data
  jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation

  Jina-reranker-v3.5 introduces a hybrid attention architecture and self-distillation protocol to improve retrieval performance across varied and semi-structured domains.

• Recovering Pruned LLM Generation Capabilities
  ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  ShortOPD uses a dynamic distillation strategy to fix structural collapse in pruned LLMs by adjusting training rollouts based on model output quality.

• Automated Skill Generation For AI Agents
  COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation

  The paper introduces COLLEAGUE.SKILL, an end-to-end workflow that distills heterogeneous traces of person-grounded knowledge into inspectable, correctable, and agent-usable skills.

• Improving Model Training with Trust Regions
  Trust-Region Behavior Blending for On-Policy Distillation

  The paper introduces a method called Trust Region Behavior Blending to stabilize on-policy distillation by controlling how teachers supervise student models during early training.