Improving Structured Pruning for LLMs

Summary & Methodology Analysis

The paper introduces Calibrated Orthogonal-Equivalence Compensation (COEC) to mitigate the performance degradation that typically occurs during structured pruning, where entire columns of weight matrices are removed to reduce inference latency. Traditional methods often rely on direct least-squares refitting of the remaining weights. However, the authors note that this approach is prone to overfitting small calibration datasets and can cause significant distortion to the original pretrained weights, leading to degraded model output quality.

To address this, COEC utilizes a methodology that alternates between a two-sided spectral update and rescaling. This involves applying an orthogonal Procrustes solution for left rotation and optimizing the right rotation on a Stiefel manifold using projected gradient descent. By applying per-mode singular-value rescaling and using generalized cross-validation to determine regularization strength independently for each layer, the method preserves the necessary correlations between weight components that are otherwise lost during standard pruning.

The authors evaluate this technique across the Llama-3, Llama-3.1, and Qwen2.5 model families. A key finding is that COEC consistently improves WikiText-2 perplexity, a measure of how well a model predicts a text sequence, at 30% column sparsity. While the method shows superior results in both perplexity and zero-shot accuracy compared to existing compensation methods, the paper notes that its effectiveness is tied to the careful calibration of the retained weights, which prevents the unwanted distortion seen in simpler refitting approaches.