• Home
  • Search
  • Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German Neurovascular Guidelines
  • Cite Icon4
  • https://doi.org/10.1007/s00062-025-01562-zCopy DOI Icon

Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German Neurovascular Guidelines

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

PurposeTo investigate the feasibility of Retrieval-augmented Generation (RAG)-enhanced Large Language Models (LLMs) in answering questions about two German neurovascular guidelines.MethodsFour LLMs (GPT-4o-mini, Llama 3.1 405B Instruct Turbo, Mixtral 8 × 22B Instruct, and Claude 3.5 Sonnet) with RAG as well as GPT-4o-mini without RAG were evaluated for generating answers about two German neurovascular guidelines (“S3 Guideline for Diagnosis, Treatment, and Follow-up of Extracranial Carotid Stenosis” and “S2e Guideline for Acute Therapy of Ischemic Stroke”). The answers were classified as “correct”, “inaccurate”, or “incorrect” by two neurovascular experts in consensus. Additionally, retrieval performance of five retrieval strategies was analyzed on a synthetic dataset of 384 questions.ResultsClaude Sonnet 3.5 achieved the highest answer correctness (70.6% correct, 10.6% wrong), followed by Llama 3.1 (64.7%, 15.3% wrong), GPT-4o-mini with RAG (57.6%, 15.3% wrong), and Mixtral (56.6%, 17.6% wrong). GPT-4o-mini without RAG performed significantly worse (20.0%, 32.9% wrong). Retrieval errors were the primary cause of incorrect answers (80%). For retrieval, BM25 achieved the highest accuracy (82.0%), outperforming vector-based methods like “BAAI/bge-m3” (78.4%).ConclusionRAG significantly improves LLM accuracy for medical guideline question answering compared to the inherent knowledge of pretrained LLMs alone while still showing significant error rates. Improved accuracy and confidence metrics are needed for safer implementation in clinical routine. Additionally, our results demonstrate the strong performance of general LLMs in medical question answering for non-English languages, such as German, even without specific training.Supplementary InformationThe online version of this article (10.1007/s00062-025-01562-z) contains supplementary material, which is available to authorized users.

Similar Papers
  • Research Article

Evaluating large language models for clinical note processing: local fine-tuning and internal-external validation using electronic health records from South Asia.

  • Feb 25, 2026
  • BMC medical informatics and decision making
  • Seyed Alireza Hasheminasab +18
  • PDF
  • Preprint Article

Knowledge Enhancement of Small-Scale Models in Medical Question Answering (Preprint)

  • Nov 03, 2024
  • Xinbai Li +3
  • Research Article

Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework

  • May 07, 2025
  • Mathematics
  • Yusong Ke +4
  • Research Article
  • Citations21

Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study

  • Apr 07, 2025
  • JMIR Formative Research
  • Denise Lee +6
  • Research Article

Medical question answering: A comprehensive multimodal and LLM-driven survey.

  • Oct 01, 2026
  • Computer methods and programs in biomedicine
  • Eya Mhedhbi +4
  • Research Article
  • Citations39

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering

  • Dec 24, 2023
  • medRxiv
  • Han Yang +5
  • Research Article
  • Citations16

Integration of Multi-Source Medical Data for Medical Diagnosis Question Answering.

  • Mar 01, 2025
  • IEEE transactions on medical imaging
  • Qi Peng +8
  • Research Article

MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering.

  • Jan 01, 2024
  • AMIA ... Annual Symposium proceedings. AMIA Symposium
  • Yucheng Shi +6
  • Research Article

BEnchmarking Large Language Models for Ophthalmology (BELO): An Expert-Curated Data Set and Evaluation Framework for Knowledge and Reasoning

  • Dec 26, 2025
  • Ophthalmology Science
  • Sahana Srinivasan +31
  • PDF
  • Research Article
  • Citations74

Performance of Large Language Models on Medical Oncology Examination Questions

  • Jun 18, 2024
  • JAMA Network Open
  • Jack B Longwell +7
  • Research Article
  • Citations52

Reasoning with large language models for medical question answering.

  • Jul 03, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Mary M Lucas +3
  • Research Article
  • Citations45

MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering

  • Jul 31, 2024
  • Artificial Intelligence In Medicine
  • Iñigo Alonso +2
  • Research Article
  • Citations3

MedARC: Adaptive multi-agent refinement and collaboration for enhanced medical reasoning in large language models.

  • Feb 01, 2026
  • International journal of medical informatics
  • Yongming Miao +3
  • Research Article
  • Citations19

Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models

  • Jul 31, 2025
  • JAMA Ophthalmology
  • Sahana Srinivasan +18
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.