Learning Through Mistakes: Explainable Medical Question-Answering using Reinforcement Learning in Large Language Models

Large Language Models (LLMs) have shown impressive performance in natural language tasks, but their outputs can exhibit undesirable attributes or biases. Existing methods for steering LLMs toward desired attributes often assume unbiased representations and rely solely on steering prompts. However, the representations learned from pre-training can introduce semantic biases that influence the steering process, leading to suboptimal results. We propose LLMGuardrail, a novel framework that incorporates causal analysis and adversarial learning to obtain unbiased steering representations in LLMs. LLMGuardrail systematically identifies and blocks the confounding effects of biases, enabling the extraction of unbiased steering representations. Experiments demonstrate LLMGuardrail's effectiveness in steering LLMs toward desired attributes while mitigating biases. Our work contributes to developing safe and reliable LLMs that align with desired attributes.

The integration of large language models (LLMs) into medical diagnostics has garnered substantial attention due to their potential to enhance diagnostic accuracy, streamline clinical workflows, and address health care disparities. However, the rapid evolution of LLM research necessitates a comprehensive synthesis of their applications, challenges, and future directions. This scoping review aimed to provide an overview of the current state of research regarding the use of LLMs in medical diagnostics. The study sought to answer four primary subquestions, as follows: (1) Which LLMs are commonly used? (2) How are LLMs assessed in diagnosis? (3) What is the current performance of LLMs in diagnosing diseases? (4) Which medical domains are investigating the application of LLMs? This scoping review was conducted according to the Joanna Briggs Institute Manual for Evidence Synthesis and adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). Relevant literature was searched from the Web of Science, PubMed, Embase, IEEE Xplore, and ACM Digital Library databases from 2022 to 2025. Articles were screened and selected based on predefined inclusion and exclusion criteria. Bibliometric analysis was performed using VOSviewer to identify major research clusters and trends. Data extraction included details on LLM types, application domains, and performance metrics. The field is rapidly expanding, with a surge in publications after 2023. GPT-4 and its variants dominated research (70/95, 74% of studies), followed by GPT-3.5 (34/95, 36%). Key applications included disease classification (text or image-based), medical question answering, and diagnostic content generation. LLMs demonstrated high accuracy in specialties like radiology, psychiatry, and neurology but exhibited biases in race, gender, and cost predictions. Ethical concerns, including privacy risks and model hallucination, alongside regulatory fragmentation, were critical barriers to clinical adoption. LLMs hold transformative potential for medical diagnostics but require rigorous validation, bias mitigation, and multimodal integration to address real-world complexities. Future research should prioritize explainable artificial intelligence frameworks, specialty-specific optimization, and international regulatory harmonization to ensure equitable and safe clinical deployment.

Large Language Models (LLMs) hold the potential for clinical task-shifting by processing unstructured clinical text, enabling tasks such as clinical concept extraction and medical question answering from electronic health records. If implemented reliably, such approaches may benefit over-burdened healthcare systems, particularly in resource-limited settings and for traditionally overlooked populations, provided that local fine-tuning is supported by appropriate clinical and technical expertise. However, this powerful technology remains largely understudied in real-world contexts, particularly in the Global South. This study aims to assess whether openly available LLMs can be used reliably for processing medical notes in real-world settings in South Asia. We used publicly available LLMs to parse de-identified clinical notes from a large electronic health records (EHR) database in Pakistan, containing hospital records for 8.2 million patients. ChatGPT (GPT-3.5) as a general-purpose LLM, and GatorTron (base), BioMegatron, BioBert and ClinicalBERT as medical LLMs were evaluated when applied to these data, after fine-tuning them with (a) publicly available clinical datasets namely Informatics for Integrating Biology & the Bedside (I2B2) and National NLP Clinical Challenges (N2C2) for medical concept extraction (MCE) and emrQA for medical question answering (MQA), and (b) the local Pakistani de-identified EHR dataset, which includes inpatient Discharge Summaries (DS) and Subjective, Objective, Assessment, and Plan (SOAP) notes, as detailed in this paper. MCE models were applied to these clinical notes using both 3-label and 9-label formats, while MQA models were applied to medical questions. Internal and external validation performance was measured for (a) and (b) using F1 score, precision, recall, and accuracy for MCE and BLEU and ROUGE-L, which measure lexical and sequence similarity, for MQA. When clinical LLMs were not fine-tuned on the local EHR dataset, their performance during external validation on local data was notably poorer compared to internal validation on the dataset used for fine-tuning, with reductions of at least 15% in F1 scores for MCE and 35% in ROUGE-L and BLEU scores for MQA tasks. This suggests potential bias and highlights the inability of the medical LLMs to reliably handle the data distribution of the local population without further fine-tuning and adaptation. This trend persisted across two distinct natural language processing tasks: concept extraction and question answering, spanning a spectrum of task complexities. However, fine-tuning the LLMs with local EHR data significantly improved model performance across both tasks, yielding a 7.5% to 15% increase in the F1 score for MCE and a 27% to 53% increase in ROUGE-L and BLEU scores for MQA. Notably, ChatGPT, as a general-purpose LLM, stood out as an exception, demonstrating superior performance across all measured metrics on the local dataset compared to the publicly available dataset, with improvements ranging from 3% to 17% on the local EHR dataset, even without fine-tuning on the local data. Publicly available LLMs, predominantly trained on data from high-income regions, were found to be unreliable when applied in a real-world clinical setting in Pakistan. Fine-tuning them with local EHR data and regional clinical contexts improved their reliability, demonstrating a feasible adaptation strategy that is substantially less resource-intensive than training large language models from scratch. Close collaboration between local clinical and technical experts to curate and leverage more representative, inclusive, and unbiased medical datasets, can play a crucial role in further ensuring reliability of LLMs for resource-limited, overburdened settings, to be used in ways that are safe, fair, and beneficial for all.

Large language models (LLMs) often struggle with factual accuracy in knowledge-intensive domains like healthcare. We introduce LEAF (Learning and Evaluation Augmented by Fact-Checking), a framework for improving LLM factuality in medical question answering. LEAF comprises three components: (1) RAFE, a robust fact-checking system using open-source LLMs and domain-specific retrieval to evaluate response accuracy; (2) Fact-Check-then-RAG, which leverages fact-checking results to guide retrieval without parameter updates; and (3) Learning from Fact Check, enabling self-training through supervised fine-tuning or preference-based learning using fact-checking as pseudo-labels. Experimental results show that RAFE outperforms Factcheck-GPT in detecting inaccuracies, Fact-Check-then-RAG effectively corrects errors, and Learning from Fact Check improves performance without labeled data. In a real-world healthcare deployment with proprietary medical documents, LEAF achieved an 83% improvement in factuality scores, demonstrating practical applicability for adapting general-purpose LLMs to organization-specific knowledge. Our framework provides a scalable solution for industrial applications requiring high factual accuracy.

Large language models (LLMs) are increasingly adopted in medical question answering (QA) scenarios. However, LLMs have been proven to generate hallucinations and nonfactual information, undermining their trustworthiness in high-stakes medical tasks. Conformal Prediction (CP) is now recognized as a robust framework within the broader domain of machine learning, offering statistically rigorous guarantees of marginal (average) coverage for prediction sets. However, the applicability of CP in medical QA remains to be explored. To address this limitation, this study proposes an enhanced CP framework for medical multiple-choice question answering (MCQA) tasks. The enhanced CP framework associates the non-conformance score with the frequency score of the correct option. The framework generates multiple outputs for the same medical query by leveraging self-consistency theory. The proposed framework calculates the frequency score of each option to address the issue of limited access to the model’s internal information. Furthermore, a risk control framework is incorporated into the enhanced CP framework to manage task-specific metrics through a monotonically decreasing loss function. The enhanced CP framework is evaluated on three popular MCQA datasets using off-the-shelf LLMs. Empirical results demonstrate that the enhanced CP framework achieves user-specified average (or marginal) error rates on the test set. Moreover, the results show that the test set’s average prediction set size (APSS) decreases as the risk level increases. It is concluded that it is a promising evaluation metric for the uncertainty of LLMs.

Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support. This potential has been illustrated by the state-of-the-art performance obtained by LLMs in Medical Question Answering, with striking results such as passing marks in licensing medical exams. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations, written by medical doctors, of the correct and incorrect options in the exams. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs, with best results around 75 accuracy for English, still has large room for improvement, especially for languages other than English, for which accuracy drops 10 points. Therefore, despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. Data, code, and fine-tuned models will be made publicly available.11https://huggingface.co/datasets/HiTZ/MedExpQA.

Large language models (LLMs) have shown impressive capabilities across various natural language tasks. However, evaluating their alignment with human preferences remains a challenge. To this end, we propose a comprehensive human evaluation framework to assess LLMs’ proficiency in following instructions on diverse real-world tasks. We construct a hierarchical task tree encompassing 7 major areas covering over 200 categories and over 800 tasks, which covers diverse capabilities such as question answering, reasoning, multiturn dialogue, and text generation, to evaluate LLMs in a comprehensive and in-depth manner. We also design detailed evaluation standards and processes to facilitate consistent, unbiased judgments from human evaluators. A test set of over 3,000 instances is released, spanning different difficulty levels and knowledge domains. Our work provides a standardized methodology to evaluate human alignment in LLMs for both English and Chinese. We also analyze the feasibility of automating parts of evaluation with a strong LLM (GPT-4). Our framework supports a thorough assessment of LLMs as they are integrated into real-world applications. We have made publicly available the task tree, TencentLLMEval dataset, and evaluation methodology which have been demonstrated as effective in assessing the performance of Tencent Hunyuan LLMs 1 . By doing so, we aim to facilitate the benchmarking of advances in the development of safe and human-aligned LLMs.

Large language models (LLMs) are rapidly advancing natural language processing and driving increasing interest across biomedical research, clinical care, and healthcare operations. This systematic review synthesizes current empirical evidence on LLM applications in healthcare and biomedical informatics, focusing on their capabilities, evaluation practices, and reported outcomes. Existing studies highlight promising uses in research, where LLMs assist literature-informed genomic interpretation, functional annotation, and biological hypothesis generation, as well as support tasks related to protein and multiomics analysis. In clinical contexts, LLMs are primarily evaluated for natural language-driven tasks, including electronic health record summarization, clinical documentation support, medical question answering, and information extraction, rather than autonomous diagnostic or therapeutic decision-making. Early investigations also describe potential value in healthcare operations, patient communication, clinician education, and drug discovery workflows largely through knowledge retrieval, text generation, and semantic search. However, current evidence remains preliminary: most studies are retrospective, benchmark-based, simulation-driven, or limited to controlled research settings. Key challenges include hallucination and unreliable reasoning, bias and inequitable performance across populations, data privacy and security constraints, reproducibility limitations, and lack of prospective clinical validation and regulatory guidance. Emerging strategies, such as domain-specific pretraining, retrieval augmentation, multi-modal architectures, and privacy-preserving learning, aim to improve reliability, safety, and real-world applicability. This review concludes by outlining methodological, infrastructural, and governance requirements needed to responsibly integrate LLMs into biomedical workflows, emphasizing that clinical deployment remains exploratory and must be rigorously evaluated.

The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.

With the rapid development of large language models (LLMs) in natural language processing and generation tasks, researchers are continuously validating their application potential in the field of medical artificial intelligence (AI). Generative Pre-trained Transformer 5 (GPT-5), Pathways Language Model 2 (PaLM-2), Large Language Model Meta AI 3 (Llama-3), Gemini-2.5, and various medical vertical models have demonstrated excellent performance in many medical and health-related tasks. However, LLMs still face key bottlenecks in real-world medical scenarios, including “illusions” slow knowledge updates, and a lack of interpretability. These problems severely restrict the safe deployment of large language models in high-risk medical scenarios and have become a major obstacle to the transition of medical artificial intelligence from research to clinical application. Retrieval-augmented generation (RAG) technology has therefore become an important solution for improving the credibility of medical LLMs. RAG significantly reduces the risk of information errors by retrieving authoritative medical knowledge before generation while maintaining knowledge updates without retraining. A review of research progress on RAG in the medical field is of great theoretical value and practical significance for promoting the design, evaluation, and standardized application of trustworthy medical AI. This review aims to systematically summarize the research progress of RAG in medical scenarios, including the technical frameworks, typical applications, and development trends of three types of methods—naive RAG, advanced RAG, and modular RAG—and further discuss their significance in medical reliability, health equity, and personalized medicine. The naive RAG architecture, through a standard “index-retrieval-generation” process, transforms external knowledge bases into a structured vector space that can be queried by an LLM, achieving positive results in electronic medical record (EMR) summary, preoperative assessment, and medical question answering (QA). However, naive RAG still has limitations in retrieval accuracy and cross-modal processing. To address these shortcomings, advanced RAG methods have been devised that significantly improve the model’s decision-making capabilities by improving retrieval strategies, enhancing inference chains, and introducing self-reflection mechanisms. Furthermore, modular RAG design offers composable, multi-module systems to support multi-source knowledge integration and complex task decomposition. In terms of application value, RAG’s contributions to the healthcare field are primarily reflected in these aspects. On the one hand, it significantly improves the reliability of medical AI by introducing traceable knowledge to reduce illusions. On the other hand, it helps promote healthcare equity by driving a shift toward a patient-centered healthcare model through localized knowledge bases and multilingual support. However, the deployment of RAG in medical scenarios still faces many challenges, such as data privacy risks. Future research should focus on improving self-supervised reflection capabilities and developing cross-modal knowledge fusion technologies. As research develops, RAG is expected to become a core foundational capability of future intelligent healthcare systems, further promoting the development of safe and efficient intelligent healthcare.

PurposeCurrent benchmarks evaluating large language models (LLMs) in ophthalmology are narrow and disproportionately prioritize accuracy. We introduce BEnchmarking LLMs for Ophthalmology (BELO), a standardized evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BEnchmarking LLMs for Ophthalmology assesses ophthalmology-related knowledge and reasoning quality.SubjectsThis study did not involve human participation.DesignCross-sectional study.MethodsUsing keyword matching and a fine-tuned PubMed Bidirectional Encoder Representations from Transformers model, we curated ophthalmology-specific multiple-choice questions (MCQs) from diverse medical data sets (Basic and Clinical Science Course [BCSC], Multi-Subject Multi-Choice Dataset for Medical domain [MedMCQA], Medical Question Answering [MedQA], Biomedical Semantic Indexing and Question Answering [BioASQ], and PubMed Question Answering [PubMedQA]). The data set underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by 3 senior ophthalmologists. To illustrate BELO's utility, we evaluated 8 LLMs (OpenAI o1, o3-mini, GPT-5, GPT-4o, DeepSeek-R1, MedGemma-4B, Llama-3-8B, and Gemini 1.5 Pro).Main Outcome MeasuresThe 8 LLMs were evaluated in terms using accuracy, macro-F1, and 5 text-generation metrics (Recall-Oriented Understudy for Gisting Evaluation, BERTScore, BARTScore, Metric for Evaluation of Translation with Explicit Ordering, and AlignScore). In a further evaluation involving human experts, 2 ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness.ResultsBEnchmarking LLMs for Ophthalmology consists of 900 high-quality, expert-reviewed questions aggregated from 5 sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). To demonstrate BELO's utility, we conducted a series of benchmarking exercises. In the quantitative evaluation, GPT-5 achieved the highest accuracy (0.90, 95% confidence interval [CI]: 0.89–0.92) and macro-F1 score (0.91, 95% CI: 0.89–0.93). On the other hand, the models' performance on text-generation metrics varied and were generally suboptimal, with scores ranging from 20.4 to 72.0 (out of 100, excluding the BARTScore metric), indicating room for improvement in clinical reasoning. In expert evaluations, GPT-4o was rated highest for accuracy and readability, while Gemini 1.5 Pro scored highest for completeness. A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO data set will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.ConclusionsBEnchmarking LLMs for Ophthalmology provides a robust clinically relevant benchmark for evaluating both the accuracy and reasoning capabilities of current and emerging LLMs in ophthalmology. Future BELO benchmarking efforts will expand to include vision-based question answering and clinical scenario management tasks.Financial Disclosure(s)The authors have no proprietary or commercial interest in any materials discussed in this article.

Despite the impressive performance across numerous tasks, Large Language Models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, Reinforcement Learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. First, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with LOw-Rank Adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. (i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. (ii) Benefiting from LLMs’ open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. (iii) Under our framework, there is no significant loss of the LLMs’ original ability during online PPO finetuning.a

Purpose This paper aims to evaluate how effectively large language models (LLMs) represent and generate minority cultural knowledge, specifically Taiwanese Hakka culture. To address this, the study proposes a structured and replicable evaluation framework integrating Bloom’s taxonomy and retrieval-augmented generation (RAG). The research is guided by the following questions: (1) How do LLMs perform across different cognitive domains when processing Hakka cultural content? (2) To what extent does the integration of RAG enhance the accuracy and contextual appropriateness of LLM outputs? And (3) How do different LLM architectures compare in their ability to recall, analyse and creatively synthesize culturally grounded information? Design/methodology/approach This study proposes a cognitive benchmarking framework to evaluate how LLMs process and apply culturally specific knowledge. The framework integrates Bloom’s taxonomy with RAG to assess model performance across six hierarchical cognitive domains: remembering, understanding, applying, analysing, evaluating and creating. Using a curated Taiwanese Hakka digital cultural archive as the primary testbed, the evaluation measures LLM-generated responses’ semantic accuracy and cultural relevance. Findings The evaluation results indicate that LLMs augmented with RAG exhibit marked improvements over baseline models in the cognitive domains of remembering, understanding and analysing. These enhancements are particularly evident in tasks requiring factual accuracy, contextual relevance and semantic precision, underscoring RAG’s effectiveness in addressing the knowledge sparsity typically observed in underrepresented cultural data sets. However, a notable limitation persists across all models including those equipped with RAG in the domain of creating. This suggests that while retrieval mechanisms bolster the reproduction and comprehension of cultural knowledge, they do not yet sufficiently support culturally nuanced generative synthesis. Originality/value This study introduces a novel evaluation framework integrating cognitive domain benchmarks with RAG-enhanced LLMs to assess cultural knowledge processing. The research advances culturally grounded artificial intelligence (AI) systems and digital archival quality by empirically demonstrating RAG’s impact on improving factual accuracy in lower and mid-level tasks. The findings affirm the strategic value of retrieval integration for enhancing representational fidelity in cultural AI applications, while also highlighting the need for future research into hybrid architectures that combine external grounding with culturally adaptive generation strategies.

A survey for large language models in biomedicine.

MedSumGraph: enhancing GraphRAG for medical QA with summarization and optimized prompts.