Reducing Computational Overhead in Multimodal Models

Summary & Methodology Analysis

OmniPack addresses the latency and throughput challenges in multimodal LLMs by reducing the token count before and during the execution of Transformer blocks, which are the neural network layers that process sequential data using attention to weigh the importance of different input parts. The pre-LLM compression stage reduces input volume by using attention and spatiotemporal cues to select important tokens, applying DPC-KNN for coverage selection based on positional distances, and using similarity-aware merging to aggregate information into representative tokens. This front-end filtering discards redundant data before it enters the expensive computation layers.

The inner-LLM compression stage performs query-conditioned collaborative compression after selected Transformer blocks. This mechanism refines the remaining audio-visual tokens based on textual relevance and cross-modal collaboration. By compressing tokens during the forward pass, the model balances data reduction with the need to maintain context-aware interactions between the audio, visual, and textual modalities. This method allows models like Qwen2.5-Omni-7B to retain 92.9% of their original performance even when restricted to 6.8% of the baseline FLOPs, representing a significant optimization for inference pipelines.

Despite these performance gains, the paper notes specific constraints regarding the placement of the inner-LLM compression layers. Applying this compression too early in the model architecture restricts the necessary multimodal interaction required for complex tasks, while applying it too late diminishes the potential computational benefits. These trade-offs define the operational envelope for deploying OmniPack, as developers must select optimal points for compression to balance latency reductions against the accuracy requirements of the specific multimodal task.