• Home
  • Search
  • Toward expert-level medical question answering with large language models
  • Cite Icon836
  • https://doi.org/10.1038/s41591-024-03423-7Copy DOI Icon

Toward expert-level medical question answering with large language models

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a ‘passing’ score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.

Similar Papers
  • Research Article

Evaluating large language models for clinical note processing: local fine-tuning and internal-external validation using electronic health records from South Asia.

  • Feb 25, 2026
  • BMC medical informatics and decision making
  • Seyed Alireza Hasheminasab +18
  • PDF
  • Preprint Article

Knowledge Enhancement of Small-Scale Models in Medical Question Answering (Preprint)

  • Nov 03, 2024
  • Xinbai Li +3
  • Research Article

Medical question answering: A comprehensive multimodal and LLM-driven survey.

  • Oct 01, 2026
  • Computer methods and programs in biomedicine
  • Eya Mhedhbi +4
  • Research Article

Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework

  • May 07, 2025
  • Mathematics
  • Yusong Ke +4
  • PDF
  • Research Article
  • Citations11

Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE

  • Apr 22, 2025
  • JAMA Network Open
  • Peter L Elkin +9
  • Research Article
  • Citations21

Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study

  • Apr 07, 2025
  • JMIR Formative Research
  • Denise Lee +6
  • Supplementary Content
  • Citations166

Large Language Models and Empathy: Systematic Review

  • Dec 11, 2024
  • Journal of Medical Internet Research
  • Vera Sorin +6
  • Research Article
  • Citations7

Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study

  • Dec 02, 2025
  • JMIR Medical Education
  • Dingqiao Wang +13
  • Research Article
  • Citations16

Integration of Multi-Source Medical Data for Medical Diagnosis Question Answering.

  • Mar 01, 2025
  • IEEE transactions on medical imaging
  • Qi Peng +8
  • Research Article

DeepSeek R1 Distilled Fails to Perform Well Against the USMLE and Other LLMs with and Without Semantics.

  • May 21, 2026
  • Studies in health technology and informatics
  • Peter L Elkin +4
  • Research Article
  • Citations53

Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions.

  • Nov 21, 2024
  • Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing
  • Guangzhi Xiong +5
  • Research Article
  • Citations39

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering

  • Dec 24, 2023
  • medRxiv
  • Han Yang +5
  • PDF
  • Research Article
  • Citations6

Reasoning-based LLMs surpass average human performance on medical social skills

  • Oct 17, 2025
  • Scientific Reports
  • Khalid Ibraheem Alohali +4
  • Supplementary Content
  • Citations21

Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks

  • Dec 01, 2025
  • Journal of Medical Internet Research
  • Eun Jeong Gong +3
  • Research Article

BEnchmarking Large Language Models for Ophthalmology (BELO): An Expert-Curated Data Set and Evaluation Framework for Knowledge and Reasoning

  • Dec 26, 2025
  • Ophthalmology Science
  • Sahana Srinivasan +31
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.