Efficient Token Pruning for Multimodal Models

Summary & Methodology Analysis

OmniLLMs often face significant inference overhead due to processing long audio and video token sequences. OmniDelta addresses this by introducing a skill-driven approach to budget allocation. Instead of uniform pruning, which can discard critical evidence while keeping redundant data, this method uses task keywords and semantic expansions to categorize intent. It then shifts the fixed token budget between audio and video based on the cosine similarity between the query and these skill-based embeddings, allowing the model to focus compute resources on the most relevant input modality for a given request. This query-aware inter-modal allocation is complemented by an intra-modal strategy that redistributes tokens within each modality based on local content complexity and temporal redundancy. By refining how tokens are selected within these dynamically allocated budgets, the method maintains higher accuracy even as it prunes the input sequence. The authors show this establishes a new accuracy-efficiency Pareto frontier across pruning ratios when tested on Qwen2.5-Omni models. However, the methodology notes that direct query-to-audio/video cosine similarity is unreliable for inter-modal allocation, as queries often express intent while modality tokens represent content. Furthermore, uniform intra-modal allocation is fundamentally suboptimal because it fails to distinguish between critical evidence and redundant content in complex temporal regions. The paper validates these improvements on four specific audio-video benchmarks, including WorldSense, AVUT, VideoMME with audio, and DailyOmni.