• Home
  • Search
  • Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems
  • Cite Icon1
  • https://doi.org/10.1109/cloud67622.2025.00029Copy DOI Icon

Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems

  • Jul 7, 2025
  • Hyungwoo Lee +7 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Recent large language models (LLMs) face increasing inference latency as input context length and model size grow. Retrieval-augmented generation (RAG) exacerbates this by significantly increasing input tokens, leading to higher computational overhead during the prefill stage and prolonged time-to-first-token (TTFT). To address this, the paper proposes using a disk-based key-value (KV) cache to reduce the prefill computational burden, thereby shortening TTFT. We also introduce a disk-based shared KV cache management system, called Shared RAG-DCache, for multi-instance LLM RAG service environments. This system leverages the locality of documents related to user queries in RAG and queueing delays in LLM inference services to proactively generate and store disk KV caches for query-related documents, sharing them across multiple LLM instances to enhance inference performance. In experiments on a single host with 2 GPUs and 1 CPU, Shared RAG-DCache achieved a 15–71 % increase in throughput and up to a 12–65 % reduction in latency, depending on the resource configuration.

Similar Papers
  • Conference Article

Dispenser: Hierarchical KV Cache Management for Efficient LLM Generative Inference

  • Dec 14, 2025
  • Beiquan Cao +3
  • Conference Article

TARDIS: A GPU-Centric KV Cache Service for Efficient LLM Inference

  • Oct 11, 2025
  • Yifan Hu +7
  • Conference Article

KVO-LLM: Boosting Long-Context Generation Throughput for Batched LLM Inference

  • Jun 22, 2025
  • Zhenyu Li +8
  • Research Article

Asymmetric KV Cache Compression using State-Aware Sparsity and Quantization

  • Jan 01, 2026
  • IEEE Transactions on Circuits and Systems for Artificial Intelligence
  • Myeongji Yun +3
  • Conference Article
  • Citations39

ZExpander

  • Apr 18, 2016
  • Xingbo Wu +5
  • Research Article
  • Citations10

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

  • Jun 17, 2025
  • Proceedings of the ACM on Management of Data
  • Hailin Zhang +7
  • Research Article

Enabling efficient low-bit quantization based on matrix product operators for KV cache compression.

  • May 01, 2026
  • Neural networks : the official journal of the International Neural Network Society
  • Jia-Qi Wang +5
  • Conference Article

RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving

  • Sep 02, 2025
  • Shengnan Yue +5
  • Research Article

Large language models versus human examinee performance on Israeli anesthesiology board examinations.

  • Mar 25, 2026
  • Scientific reports
  • Ariel Ronen +10
  • Research Article
  • Citations7

Legal aspects of generative artificial intelligence and large language models in examinations and theses.

  • Jan 01, 2024
  • GMS journal for medical education
  • Maren März +3
  • Research Article
  • Citations20

DialogueLLM: Context and emotion knowledge-tuned large language models for emotion recognition in conversations.

  • Dec 01, 2025
  • Neural networks : the official journal of the International Neural Network Society
  • Yazhou Zhang +6
  • Research Article
  • Citations9

A survey for large language models in biomedicine.

  • Dec 01, 2025
  • Artificial intelligence in medicine
  • Chong Wang +16
  • Research Article

A Systematic Review of Vulnerability Scanning Tools for Large Language Models (LLMs)

  • Dec 11, 2025
  • International Journal of Science and Research (IJSR)
  • Karthikeyan Thirumalaisamy
  • Research Article

A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference

  • Mar 12, 2026
  • ACM Transactions on Design Automation of Electronic Systems
  • Chenyi Wen +5
  • Preprint Article

On the Dependable Operation of Key-Value Caches in Large Language Models (LLMs)

  • Mar 28, 2025
  • Zhen Gao +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.