Scaling KV Cache Beyond Memory Limits
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 5 concepts
Key Takeaways
- Overcomes the HBM capacity bottleneck by offloading KV cache data to host or remote DRAM.
- Implements look-ahead attention to predict and prefetch required KV blocks before they are needed for decoding.
- Maintains model performance within 0.7 points of full attention.
- Integrates into the existing vLLM architecture through a custom head-wise mapping layer.
- Uses a capped eviction policy to prevent interconnect congestion during prefetching.
Summary & Methodology Analysis
OasisKV addresses the memory bottleneck in large language model inference where the KV cache, the memory used to store transformer attention key and value states, exceeds high bandwidth memory (HBM) capacity. The system uses a foreground process to perform standard attention computation on a sparse set of KV entries currently resident in HBM. Simultaneously, a background pipeline performs look-ahead attention using draft tokens generated via speculative decoding to predict future KV block requirements. These predicted blocks are then prefetched from host or remote DRAM into HBM asynchronously, with a capped eviction policy ensuring data movement stays within bandwidth limits.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What problem does OasisKV solve?
It solves the memory wall issue in inference where limited HBM capacity restricts decode batch sizes and overall throughput during long-context and agentic workloads.
Q2. How does this improve system efficiency?
By offloading the KV cache to higher-capacity memory and only bringing the most relevant blocks into HBM, the system handles larger workloads than HBM capacity would normally allow.
Q3. Does this method degrade model performance?
No, the prediction accuracy keeps model performance within 0.7 points of full attention.
Q4. What is the role of the head-wise logical-to-logical mapping layer?
This layer manages sparse block residency for different KV heads and is integrated into the PagedAttention structure of the vLLM engine.
Q5. How are relevant KV blocks identified for prefetching?
The system utilizes draft tokens from speculative decoding to predict which KV blocks will be needed for the next decode step.
Q6. Does the system support disaggregated setups?
Yes, remote partial fetching allows for an initial sparse working set at admission with additional blocks prefetched over the network during decoding.
Q7. What happens if too many blocks are prefetched?
A capped eviction policy at the KV-selection stage bounds the number of newly admitted blocks to ensure data movement stays within the interconnect's bandwidth budget.
Q8. What are the current limitations of the prototype?
The current version does not perform speculative decoding using accepted tokens, and it assumes that request KV caches remain in the prefill node's host DRAM until completion.
Q9. Which models and benchmarks were used in the evaluation?
The paper evaluated models including EAGLE-3, Qwen3-8B, Qwen3-235B-A22B, and Llama-3.1-8B-Instruct, using benchmarks such as AIME24, AIME25, GPQA-Diamond, and LongBench v2.