HYPER-RAG: Evaluating Hyperparameter Trade-Offs in Biomedical Retrieval-Augmented Generation

Abstract

Translate Article

Retrieval-Augmented Generation (RAG) improves the factual accuracy of large language models by combining document retrieval with text generation. In biomedical question answering, where correctness is critical, the effect of key hyperparameters has not been studied in a systematic way. This paper presents an evaluation of RAG on the COVID-QA dataset with a focus on three retrievers (dense, BM25, hybrid), two retrieval depths (top- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{k}=1,3$</tex>), and optional reranking with a cross encoder. We use a single biomedical prompt and measure exact match (EM), F1 score, semantic similarity, groundedness, and latency. We also report a composite score that balances lexical accuracy, semantic similarity, and efficiency. Our results on a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{1 0 0}$</tex>-question subset show that reranking improves grounding at the cost of extra latency, and that increasing top-k improves recall but gives smaller gains after a point. The study highlights that multiple metrics are needed to judge biomedical RAG systems reliably and that careful tuning of retrieval and reranking settings can yield practical improvements under compute constraints.