- Conference Article
- 10.1109/icpads67057.2025.11323033
Dispenser: Hierarchical KV Cache Management for Efficient LLM Generative Inference
- Dec 14, 2025
- Beiquan Cao + 3 more +3
As context windows expands, Large Language Models (LLMs) are being more capable of handling long-context tasks. However, long-context inference is hindered by high latency and memory usage due to the linearly growing Key-Value (KV) cache. Prior work has shown that a dynamic subset of the KV cache dominates attention outcomes and selectively loads these subsets instead of the full cache. These methods partition the KV cache into segments (block-wise and token-wise) and identify critical ones for recall. However, for block-wise ones, their recall accuracy is often compromised by semantic inconsistencies among consecutive tokens, and for token-wise ones, they suffer from inefficiency in memory management and offloading. To these ends, we propose Dispenser, a hierarchical KV cache management algorithm inspired by multi-level page tables. Dispenser organizes the KV cache based on token semantic similarity and partitions it into pages and blocks, which enables straightforward integration into modern serving engines such as vLLM. It employs a two-level lookup mechanism to identify and load the most relevant blocks for the current query. Experimental results demonstrate that Dispenser achieves up to a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5.75 \times$</tex> speedup in self-attention, reducing inference latency by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.1 \times$</tex> on tasks with long dependencies with negligible accuracy loss.
Read more