Efficient Caching for Multimodal LLM Serving

Summary & Methodology Analysis

The paper introduces QV-PIC, a solution for the quality degradation seen in rendered-image position-independent caching. The pipeline begins with offline template-conditioned cache preparation. Text chunks are rendered at multiple resolutions, and the system compiles KV caches (the stored internal key-value pairs representing context in transformers) under a chat-template prefix before stripping those entries to ensure compatibility. Source texts are indexed using BGE-M3 embeddings to support later ranking by query relevance. This approach solves the loss of fine-grained textual information common in previous visual compression methods without requiring expensive online repairs. The online stage features query-aware cache assembly, which scores retrieved chunks against incoming queries. Based on a cumulative relevance threshold and a set budget, the system promotes specific chunks to high-resolution versions. Finally, the framework employs M-RoPE (a modification to Rotational Positional Embeddings that track sequence order) re-anchoring to adjust cache keys, ensuring they align with the request positions in the current context order. While effective, the paper notes that uniform DPI scaling for rendered images leads to inconsistent F1 improvements while reliably increasing system latency.