Recovering Pruned LLM Generation Capabilities

Summary & Methodology Analysis

Structurally compressed models frequently suffer from a recognition-generation gap, where they retain accuracy on static benchmarks but fail during free-form generation by producing repetitive, low-information text. This happens because standard compression techniques often rely on off-policy training targets that do not align with the sequential nature of autoregressive generation. To address this, the authors introduce ShortOPD, which utilizes on-policy distillation, a process where a student model generates its own sequences and is forced to match the token probability distribution of a frozen, pre-compression teacher model using a generalized Jensen-Shannon divergence loss. By aligning the student's internal logic with the teacher during active generation, the model regains the ability to produce coherent, non-repetitive sequences.

The core of the strategy is a short-to-long schedule that dynamically modulates the per-step rollout budget. A gated controller monitors training outputs for repetition or clean truncation, adjusting the amount of generated content the model uses to update its weights. If the model exhibits high repetition, the budget is shrunk to force it back toward stable patterns; if the model produces clean text, the budget is expanded to encourage better long-form generation. These updates are smoothed using exponential moving averages to prevent unstable oscillations in the model parameters during the training process. The system was validated across various benchmarks including Qwen3-4B-Instruct, GSM8K, MATH, HumanEval, MBPP, and others.

While effective at restoring performance, the method has clear bounds. The paper does not specify the exact threshold of structural compression beyond which a model becomes impossible to recover using ShortOPD. Engineering teams should be aware that while this technique mitigates the degradation associated with pruning, it does not guarantee a return to the original model's baseline performance, as evidenced by the recovery of approximately two-thirds of the original generation score. The technique remains a post-processing step for models that have already undergone structural pruning.