• Home
  • Search
  • Which curriculum components do medical students find most helpful for evaluating AI outputs?
  • Cite Icon8
  • https://doi.org/10.1186/s12909-025-06735-5Copy DOI Icon

Which curriculum components do medical students find most helpful for evaluating AI outputs?

Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

IntroductionThe risk and opportunity of Large Language Models (LLMs) in medical education both rest in their imitation of human communication. Future doctors working with generative artificial intelligence (AI) need to judge the value of any outputs from LLMs to safely direct the management of patients. We set out to investigate medical students’ ability to evaluate LLM responses to clinical vignettes, identify which prior learning they utilised to scrutinise the LLM answers, and assess their awareness of ‘clinical prompt engineering’.MethodsFinal year medical students were asked in a survey to assess the accuracy of the answers provided by generative pre-trained transformer (GPT) 3.5 in response to ten clinical scenarios, five of which GPT 3.5 had answered incorrectly, and to identify which prior training enabled them to evaluate the GPT 3.5 output. A content analysis was conducted amongst 148 consenting medical students.ResultsThe median percentage of students who correctly evaluated the LLM output was 56%. Students reported interactive case-based and pathology teaching using questions to be the most helpful training provided by the medical school for evaluating AI outputs. Only 5% were familiar with the concept of ‘clinical prompt engineering’.ConclusionPathology and interactive case-based teaching using questions were the self-reported best training for medical students to safely interact with the outputs of LLMs. This study can inform the design of medical training for future doctors graduating into AI-enhanced health services.

Loading PDF

Similar Papers
  • Research Article
  • Citations14

The general intelligence of GPT–4, its knowledge diffusive and societal influences, and its governance

  • Mar 28, 2024
  • Meta-Radiology
  • Mohammad Mahdi Jahani Yekta
  • Research Article

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

  • Nov 03, 2025
  • Blood
  • Samantha Rizzo +5
  • Research Article
  • Citations39

A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions Like We Do?

  • Jan 31, 2025
  • Manufacturing & Service Operations Management
  • Yang Chen +4
  • Research Article
  • Citations1

GPT-4o and OpenAI o1 Performance on the 2024 Spanish Competitive Medical Specialty Access Examination: Cross-Sectional Quantitative Evaluation Study

  • Jan 12, 2026
  • JMIR Medical Education
  • Pau Benito +9
  • PDF
  • Research Article
  • Citations126

Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department

  • May 07, 2024
  • JAMA Network Open
  • Christopher Y K Williams +6
  • Research Article

Performance of three commercially available large language models and one locally fine-tuned model at preparing formal letters to appeal medical insurance denials of radiotherapy services.

  • Jun 01, 2024
  • Journal of Clinical Oncology
  • Kendall Kiser +4
  • Research Article
  • Citations1

Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation

  • Nov 20, 2025
  • JMIR Formative Research
  • Alexandre Hudon +3
  • PDF
  • Research Article
  • Citations26

Large language models for error detection in radiology reports: a comparative analysis between closed-source and privacy-compliant open-source models

  • Feb 20, 2025
  • European Radiology
  • Babak Salam +12
  • Research Article
  • Citations24

A dataset and benchmark for hospital course summarization with adapted large language models.

  • Dec 30, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Asad Aali +11
  • Research Article

Generative AI in Medical Pharmacology: Balancing Educational Benefits and Hallucination Risks

  • Apr 19, 2025
  • International Journal of Science and Research (IJSR)
  • Swati Rai +2
  • Research Article

Integrating Large Language Models Into Trauma Education for Medical Students: Randomized Controlled Pilot Trial

  • Mar 17, 2026
  • JMIR Medical Education
  • Joona Gustafsson +2
  • Research Article
  • Citations11

How do generative artificial intelligence (AI) tools and large language models (LLMs) influence language learners’ critical thinking in EFL education? A systematic review

  • Aug 04, 2025
  • Smart Learning Environments
  • Jing Liu +2
  • Research Article
  • Citations136

Vision-Language Models in Remote Sensing: Current progress and future trends

  • Jun 01, 2024
  • IEEE Geoscience and Remote Sensing Magazine
  • Xiang Li +4
  • Research Article

Transformative Trends: A Comprehensive Review of Large Language Models (LLMs) in Healthcare

  • Jun 02, 2024
  • INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
  • Chetna Kumari
  • Research Article

Large language model use in oral and maxillofacial surgery training: a national resident survey.

  • Feb 21, 2026
  • Oral and maxillofacial surgery
  • Nolan Kranc +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.