• Home
  • Search
  • RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving
  • https://doi.org/10.1145/3748273.3749196Copy DOI Icon

RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving

  • Sep 2, 2025
  • Shengnan Yue +5 more
Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Modern large language model (LLM) serving systems increasingly adopt a prefill-decode disaggregation architecture to enhance inference efficiency. While this design improves resource utilization, it introduces latency due to the transfer of key-value (KV) cache. The community has generally assumed that this latency is bandwidth-bound and can be effectively mitigated by high-speed interconnects.

Similar Papers
  • Conference Article
  • Citations1

Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems

  • Jul 07, 2025
  • Hyungwoo Lee +7
  • Conference Article

Dispenser: Hierarchical KV Cache Management for Efficient LLM Generative Inference

  • Dec 14, 2025
  • Beiquan Cao +3
  • Conference Article

TARDIS: A GPU-Centric KV Cache Service for Efficient LLM Inference

  • Oct 11, 2025
  • Yifan Hu +7
  • Research Article

Asymmetric KV Cache Compression using State-Aware Sparsity and Quantization

  • Jan 01, 2026
  • IEEE Transactions on Circuits and Systems for Artificial Intelligence
  • Myeongji Yun +3
  • Conference Article

KVO-LLM: Boosting Long-Context Generation Throughput for Batched LLM Inference

  • Jun 22, 2025
  • Zhenyu Li +8
  • Research Article

Unlocking the Potential of Large Language Models in Education: Factors Influencing Adoption by Instructional Designers and Academics

  • Jan 01, 2026
  • Journal of Information Technology Education: Research
  • Katherine L Fourie +2
  • Conference Article
  • Citations39

ZExpander

  • Apr 18, 2016
  • Xingbo Wu +5
  • Research Article

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

  • Nov 03, 2025
  • Blood
  • Samantha Rizzo +5
  • Supplementary Content

Large language and vision-language models for robot: safety challenges, mitigation strategies and future directions

  • Jul 29, 2025
  • Industrial Robot: the international journal of robotics research and application
  • Xiangyu Hu +1
  • Research Article
  • Citations10

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

  • Jun 17, 2025
  • Proceedings of the ACM on Management of Data
  • Hailin Zhang +7
  • Research Article

Research and selection of Large Learning Models for automation of ABAP-code migration

  • Sep 24, 2025
  • Management of Development of Complex Systems
  • Oleg Pozdnyakov +1
  • Research Article

Readability & quality of large language model responses in CAR-T patient education

  • Nov 03, 2025
  • Blood
  • Sridhar Balasubramanian +3
  • Research Article

Application of Large Language Models (LLMs) to Geriatric Practice and Its Evaluation at 4 VA GRECCs

  • Dec 01, 2025
  • Innovation in Aging
  • Huai Cheng +2
  • Research Article

1401 Bringing Medicine Expertise to Your Screen: A New Frontier in Curbside Sleep Consultation Leveraging Large Language Models?

  • May 19, 2025
  • SLEEP
  • Nina Kuei +4
  • Research Article

Enabling efficient low-bit quantization based on matrix product operators for KV cache compression.

  • May 01, 2026
  • Neural networks : the official journal of the International Neural Network Society
  • Jia-Qi Wang +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.