KV cache

What it is

During inference, an LLM processes inputs token by token. For each new token, the model needs to reference the Key and Value matrices derived from all preceding tokens in the sequence. By caching these matrices in GPU VRAM instead of discarding them, the system avoids redundant matrix multiplications. This cache consumes memory proportional to sequence length and hidden dimension size, often reaching several gigabytes for long context windows.

Why it matters

The size of the KV cache is the primary bottleneck for concurrent user throughput and maximum context length. If you ignore its memory footprint, your instances will encounter out-of-memory errors as concurrent sessions grow. Managing this cache effectively determines whether your infrastructure is cost-efficient or prone to catastrophic latency spikes when handling long documents.

In practice

You manage the KV cache through inference engine settings like PagedAttention or memory management flags in frameworks like vLLM or TGI. You will observe its impact when monitoring GPU utilization metrics or setting the maximum sequence length parameter in your API calls to prevent memory exhaustion.

The tradeoff

Expanding the context window increases memory usage linearly or quadratically depending on the architecture, forcing a direct choice between supporting longer sessions and increasing your total number of concurrent requests.

Where it appears

Research summaries that use KV cache, each linked to its source paper.

• Faster LLM Reasoning via Prefix Sliding
  Prefix Sliding for efficient test-time scaling

  Prefix Sliding enables large language models to perform reasoning tasks three times faster without requiring additional training.

• Securing Large Language Models with Semantic Overlays
  Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

  Semantic Overlays add an out-of-band annotation layer to input streams to help models distinguish between trusted instructions and untrusted content, significantly reducing prompt injection success rates.

• Improving KV Cache Eviction Using Sigmoid Gates
  Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

  The paper demonstrates that using learned sigmoid gates for key-value cache eviction leads to better performance than existing methods like H2O and KeyDiff.

• Improving Rule Reasoning in Large Models
  Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

  The paper introduces a framework called DynaRule that enables large language models to dynamically retrieve and apply reusable procedural rules at scale.

• Accelerating Long Context LLM Inference
  FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

  FlashPrefill V2 improves large language model serving performance by introducing a block-sparse attention mechanism that integrates natively with paged KV cache and continuous batching systems.

• Optimizing Rotations for 4-Bit Model Quantization
  When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

  The researchers investigated whether using specific rotational transforms that respect RoPE structure improves accuracy during 4-bit model quantization.

• Improving Long-Term Memory in Streaming Video Models
  StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs

  StreamTTT introduces a dual-branch architecture that combines real-time attention with a recurrent state to improve video memory recall without sacrificing performance.

• Optimizing LLM Cache Transfer for Edge Handover
  ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

  ImpactHO improves LLM performance during user handovers by intelligently prioritizing and transferring essential parts of the KV cache over constrained network links.

• Nesting Language Models for Efficiency
  Matryoshka Language Model Suites

  The paper introduces a framework for nesting smaller sub-models within a larger architecture to reduce training compute and improve speculative decoding performance.

• Scaling KV Cache Beyond Memory Limits
  OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

  OasisKV uses predictive prefetching to move necessary KV cache data from high capacity memory to HBM, allowing for larger decode batches and longer context without hitting the memory wall.

• Optimizing RAG Latency via Cached Information Nuggets
  CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG

  CoinRAG reduces RAG latency and computational redundancy by precomputing and reusing specific information nugget representations within the model KV cache.

• Pruning Tool Outputs for Coding Agents
  SWE-Pruner Pro: The Coder LLM Already Knows What to Prune

  SWE-Pruner Pro automatically prunes redundant tool outputs by analyzing internal model states to lower token costs and improve performance without external scoring models.

• Xiaomi-Robotics-1: Scaling Robot Action Models
  Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories

  Xiaomi-Robotics-1 introduces a large-scale vision-language-action model and a scalable auto-labeling pipeline to overcome data bottlenecks in robotics, achieving strong performance on robot benchmarks.

• Efficient Memory Management for MoE LLMs
  PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

  PagedWeight manages GPU memory for Mixture-of-Experts models by dynamically quantizing weights at runtime to balance model precision against KV cache requirements.

• Optimizing Robot Control Through Action Sensitive Compression
  Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models

  The authors introduce a method to compress token data in vision-language-action models by identifying and prioritizing information that has the least impact on physical robot movements.

• Advancing Open Weight Desktop GUI Agents
  UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

  UI-Mate-27B improves desktop automation performance through in-context demonstrations and specialized training on task-driven benchmarks.

• Streaming Identity Swapping for Talking Videos
  UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

  UniSwap is a framework designed for low-latency, streaming-ready audio-visual identity swapping that preserves source motion and content while replacing appearance and voice.

• Real-Time Streaming Human Animation System
  LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time

  LiveAnimate is a diffusion-based framework that enables stable long-form human animation for real-time applications by optimizing architecture and inference processes.

• Optimizing Reasoning Through Thought Level Search
  Thought-Level Beam Search for Reasoning

  Gambit improves the accuracy of large reasoning models by dynamically reallocating computational resources to promising branches during inference.

• Optimizing Mixed RL Rollout Throughput
  Scheduling Mixed RL Rollouts Beyond Prefix Locality

  The paper introduces a scheduling method that increases rollout throughput and reduces iteration time by managing how heterogeneous reinforcement learning workloads share KV-cache capacity.

• Improving Long Video Memory Persistence
  Addressable Memory for Video World Models

  WorldTrace addresses long-horizon visual memory in video models by using a fixed-size cache that prevents positional embedding degradation.