KV cache

What it is

During inference, an LLM processes inputs token by token. For each new token, the model needs to reference the Key and Value matrices derived from all preceding tokens in the sequence. By caching these matrices in GPU VRAM instead of discarding them, the system avoids redundant matrix multiplications. This cache consumes memory proportional to sequence length and hidden dimension size, often reaching several gigabytes for long context windows.

Why it matters

The size of the KV cache is the primary bottleneck for concurrent user throughput and maximum context length. If you ignore its memory footprint, your instances will encounter out-of-memory errors as concurrent sessions grow. Managing this cache effectively determines whether your infrastructure is cost-efficient or prone to catastrophic latency spikes when handling long documents.

In practice

You manage the KV cache through inference engine settings like PagedAttention or memory management flags in frameworks like vLLM or TGI. You will observe its impact when monitoring GPU utilization metrics or setting the maximum sequence length parameter in your API calls to prevent memory exhaustion.

The tradeoff

Expanding the context window increases memory usage linearly or quadratically depending on the architecture, forcing a direct choice between supporting longer sessions and increasing your total number of concurrent requests.

Where it appears

Research summaries that use KV cache, each linked to its source paper.

• Recycling LLM Cache for Faster Inference
  Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

  The researchers developed a linear mapping technique to transfer and reuse KV cache data between different models in the same family, effectively skipping the expensive prefill computation step.

• Efficient Robot Vision using Fibonacci Patterns
  FibVLA: An Efficient Temporal Vision-Language-Action Model with Fibonacci Sampling

  The researchers developed a new vision model that uses a mathematical sequence to efficiently process visual history for faster and more accurate robot control.