Analyzing Bias in Large Language Models: A Quantitative Study Using Sentiment and Demographic Metrics

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

Background/Objective: Hypertension is the most prevalent chronic disease in primary care and a leading cause of cardiovascular morbidity and mortality. Despite existing guidelines, therapeutic inertia and suboptimal control persist. Large language models (LLMs) offer a potential valuable addition to augment clinical decision-making, yet their reliability for guideline-driven tasks remains unverified. This study evaluated the accuracy and safety of hypertension management recommendations generated by three LLMs compared to expert responses. Methods: Fifty-one clinical vignettes representing 17 core hypertension management concepts were constructed by hypertension experts. Each case was submitted to three LLMs (GPT-4, Gemini, MedLM) and a hypertension expert also wrote the “gold standard” answers. Three blinded expert reviewers rated each response on a 4-point accuracy scale, a binary safety (safe/unsafe) scale, and attempted to identify the source (LLM vs. expert) providing the response. Ratings were analyzed using mean scores, percentages of accurate and safe responses, and inter-rater agreement. Results: GPT-4 had the highest accuracy (83%) and safety (86%) scores among LLMs but remained inferior to expert responses (92% accuracy, 93% safety). Gemini and MedLM performed significantly worse (accuracy: 64% and 35%; safety: 73% and 39%, respectively). GPT-4 generated the most guideline-concordant responses (46%) among the three LLMs (Gemini 35%, MedLM 14%), but remains lower than experts’ responses (68%). Evaluators misidentified LLM responses as expert-written in 10 to 25% of cases, particularly with GPT-4. Inter-rater reliability for accuracy ratings was highest for expert-generated responses (ICC 0.81), with progressively lower agreement for GPT-4 (0.76), Gemini (0.70), and MedLM (0.68). A similar pattern was observed for safety and source discrimination ratings. The agreement was strongest for safety assessments and weakest for source discrimination. Conclusion: Among three tested LLMs, GPT-4 demonstrated closer agreement to expert decisions thereby showing greater potential for supporting hypertension management. However, current LLMs’ versions frequently produce inaccurate or unsafe recommendations and remain inferior to expert judgment. Human-in-the-loop supervision remains essential when deploying LLMs for clinical decision-making.

The emergence of large language models (LLMs) and the increasing prevalence of electronic health records (EHRs) present significant opportunities for advancing health care research and practice. However, research that compares and applies LLMs to extract key epilepsy-related information from unstructured medical free text is under-explored. This study fills this gap by comparing and applying different open-source LLMs and methods to extract epilepsy information from unstructured clinic letters, thereby optimizing EHRs as a resource for the benefit of epilepsy research. We also highlight some limitations of LLMs. Employing a dataset of 280 annotated clinic letters from King's College Hospital, we explored the efficacy of open-source LLMs (Llama and Mistral series) for extracting key epilepsy-related information, including epilepsy type, seizure type, current anti-seizure medications (ASMs), and associated symptoms. The study used various extraction methods, including direct extraction, summarized extraction, and contextualized extraction, complemented by role-prompting and few-shot prompting techniques. Performance was evaluated against a gold standard dataset, and was also compared to advanced fine-tuned models and human annotations. Llama 2 13b (a 13-billion-parameter LLM developed by Meta) demonstrated superior extraction capabilities across tasks by consistently outperforming other LLMs (F1 = .80 in epilepsy-type extraction, F1 = .76 in seizure-type extraction, and F1 = .90 in current ASMs extraction). Here, F1 score is a balanced metric indicating the model's accuracy in correctly identifying relevant information without excessive false positives. The study highlights the direct extraction showing consistent high performance. Comparative analysis showed that LLMs outperformed current approaches like MedCAT (Medical Concept Annotation Tool) in extracting epilepsy-related information (.2 higher in F1). The results affirm the potential of LLMs in medical information extraction relating to epilepsy, offering insights into leveraging these models for detailed and accurate data extraction from unstructured texts. The study underscores the importance of method selection in optimizing extraction performance and suggests a promising avenue for enhancing medical research and patient care through advanced natural language processing technologies.

8577 Background: Accurate and timely assessment of immunotherapy response is vital for optimizing lung cancer management. This study evaluates the efficacy of a large language model (LLM), Gemini 1.5 Pro, in automating response assessment using positron emission tomography/computed tomography (PET/CT) reports based on the European Organization for Research and Treatment of Cancer (EORTC) criteria. Methods: Google Gemini 1.5 Pro was selected due to its large context window capacity and its free availability via the web interface.The model was utilized with explicit instructions on applying EORTC criteria and fine-tuned using few-shot prompting. Pre- and post-immunotherapy PET-CT reports in text format from 33 lung cancer patients, anonymized in compliance with HIPAA regulations, were independently classified by the LLM and an experienced nuclear medicine specialist. Performance metrics, including precision, recall, F1-score, and support, were calculated for each response category. Inter-rater agreement was assessed using Cohen's Kappa. Results: The nuclear medicine specialist classified 5, 21, 6, and 1 cases as complete metabolic response (CMR), progressive metabolic disease (PMD), partial metabolic response (PMR), and stable metabolic disease (SMD), respectively, while Gemini 1.5 Pro classified 5, 20, 7, and 1 cases accordingly. The LLM achieved an overall accuracy of 97% and demonstrated excellent agreement with the expert (overall Cohen's Kappa: 0.945). F1-scores were 1.00 for CMR and SMD, 0.98 for PMD, and 0.92 for PMR, with per-label Kappa scores ranging from 0.904 (PMR) to 1.00 (CMR and SMD) (Table 1). Conclusions: Gemini 1.5 Pro exhibits strong potential for automating accurate immunotherapy response assessment in lung cancer using PET-CT reports. Its high concordance with expert evaluations underscores its utility in streamlining clinical workflows and improving efficiency. Validation with larger, more diverse datasets is warranted to support its clinical implementation. Performance metrics of Gemini 1.5 Pro for immunotherapy response assessment. Response Precision Recall F1-score Support Cohen’s Kappa CMR 1.00 1.00 1.00 5 1.000 PMD 1.00 0.95 0.98 21 0.936 PMR 0.86 1.00 0.92 6 0.904 SMD 1.00 1.00 1.00 1 1.000 CMR, Complete Metabolic Response; PMD, Progressive Metabolic Disease; PMR, Partial Metabolic Response; SMD, Stable Metabolic Disease.

Artificial intelligence (AI), and in particular large language models (LLMs) such as Chat Generative Pre-Trained Transformer (ChatGPT) and Gemini have provided additional resources for patients to research the management of healthcare conditions, for their own edification and the advocacy in the care of their children. The accuracy of these models, however, and the sources from which they draw conclusions, have been largely unstudied in pediatric orthopaedics. This research aimed to assess the reliability of machine learning tools in providing appropriate recommendations for the care of common pediatric orthopaedic conditions. ChatGPT and Gemini were queried using plain language generated from the American Academy of Orthopaedic Surgeons (AAOS) Clinical Practice Guidelines (CPGs) listed on the Pediatric Orthopedic Society of North America (POSNA) web page. Two independent reviewers assessed the accuracy of the responses, and chi-square analyses were used to compare the 2 LLMs. Inter-rater reliability was calculated via Cohen's Kappa coefficient. If research studies were cited, attempts were made to assess their legitimacy by searching the PubMed and Google Scholar databases. ChatGPT and Gemini performed similarly, agreeing with the AAOS CPGs at a rate of 67% and 69%. No significant differences were observed in the performance between the 2 LLMs. ChatGPT did not reference specific studies in any response, whereas Gemini referenced a total of 16 research papers in 6 of 24 responses. 12 of the 16 studies referenced contained errors and either were unable to be identified (7) or contained discrepancies (5) regarding publication year, journal, or proper accreditation of authorship. The LLMs investigated were frequently aligned with the AAOS CPGs; however, the rate of neutral statements or disagreement with consensus recommendations was substantial and frequently contained errors with citations of sources. These findings suggest there remains room for growth and transparency in the development of the models which power AI, and they may not yet represent the best source of up-to-date healthcare information for patients or providers.

Large Language Models (LLMs) excel in code-related tasks like code generation, but benchmark evaluations often overlook task characteristics, such as difficulty. Moreover, benchmarks are usually built using tasks described with a single prompt, despite the formulation of prompts having a profound impact on the outcome. This paper introduces a generalist approach, TaskEval, a framework using diverse prompts and Item Response Theory (IRT) to efficiently assess LLMs’ capabilities and benchmark task characteristics, improving the understanding of their performance. Using two code generation benchmarks, HumanEval + and ClassEval , as well as 8 code generation LLMs, we show that TaskEval is capable of characterising the properties of tasks. Using topic analysis, we identify and analyse the tasks of 17 and 21 topics within the benchmarks. We also cross-analyse tasks’ characteristics with programming constructs (e.g., variable assignment, conditions, etc.) used by LLMs, emphasising some patterns with tasks’ difficulty. Finally, we conduct a comparison between the difficulty assessment of tasks by human annotators and LLMs. Orthogonal to current benchmarking evaluation efforts, TaskEval can assist researchers and practitioners in fostering better assessments of LLMs. The tasks’ characteristics can be used to identify shortcomings within existing benchmarks or improve the evaluation of LLMs.

Peripheral artery disease (PAD) is a global health challenge. Advances in artificial intelligence (AI), such as large language models (LLMs) and chain-of-thought (CoT) reasoning, offer novel approaches for clinical recommendations. This study compared the readability and guideline adherence of responses from physicians and AI for a standardized PAD case. This cross-sectional study gathered responses from 30 specialized physicians (11 cardiologists, 19 vascular surgeons) across seven Latin American countries and 13 LLM systems (10 standard, three CoT). Both groups addressed diagnosis, treatment, risks, and prognosis; LLMs responded as vascular specialists. Responses were blindly evaluated with five validated Spanish readability indices and compared to the 2024 ACC/AHA multisocietal PAD guideline. Three experts scored guideline adherence; nonparametric tests were applied. Guideline adherence did not differ significantly between physicians (median 5.8 [3.4-7.6]) and LLMs (7.3 [4.7-9.7], p = 0.169), though CoT-LLMs achieved the highest scores (9.7 [8.5-11.0]). LLMs more often recommended supervised exercise (84.6% vs 30.0%, p = 0.002) and revascularization for quality of life (69.2% vs 20.0%, p = 0.004), whereas physicians favored cilostazol (60.0% vs 30.8%, p = 0.104). LLM responses had lower Readability μ values (46.9 vs 51.4, p = 0.012). Inter-rater reliability was highest for CoT-LLMs (intraclass correlation coefficient [ICC] = 0.98) versus physicians (ICC = 0.76). LLM showed comparable guideline adherence to physicians although CoT models achieved the highest scores. The difference in physician and AI treatment preferences suggest the potential of AI as adjunct clinical tools and warrants further study.

Synthetic electronic health records (EHRs) generated by large language models (LLMs) offer potential for clinical education and model training while addressing privacy concerns. However, performance variations and demographic biases in these models remain underexplored, posing risks to equitable health care. This study aimed to systematically assess the performance of various LLMs in generating synthetic EHRs and to critically evaluate the presence of gender and racial biases in the generated outputs. We focused on assessing the completeness and representativeness of these EHRs across 20 diseases with varying demographic prevalence. A framework was developed to generate 140,000 synthetic EHRs using 10 standardized prompts across 7 LLMs. The electronic health record performance score (EPS) was introduced to quantify completeness, while the statistical parity difference (SPD) was proposed to assess the degree and direction of demographic bias. Chi-square tests were used to evaluate the presence of bias across demographic groups. Larger models exhibited superior performance but heightened biases. The Yi-34B achieved the highest EPS (96.8), while smaller models (Qwen-1.8B: EPS=63.35) underperformed. Sex polarization emerged: female-dominated diseases (eg, multiple sclerosis) saw amplified female representation in outputs (Qwen-14B: 973/1000, 97.3% female vs 564,424/744,778, 75.78% real; SPD=+21.50%), while balanced diseases and male-dominated diseases skewed the male group (eg, hypertension Llama 2-13 B: 957/1000, 95.7% male vs 79,540,040/152,466,669, 52.17% real; SPD=+43.50%). Racial bias patterns revealed that some models overestimated the representation of White (eg, Yi-6B: mean SPD +14.40%, SD 16.22%) or Black groups (eg, Yi-34B: mean SPD +14.90%, SD 27.16%), while most models systematically underestimated the representation of Hispanic (average SPD across 7 models is -11.93%, SD 8.36%) and Asian groups (average SPD across 7 models is -0.77%, SD 11.99%). Larger models, such as Yi-34B, Qwen-14B, and Llama 2 to 13 B, showed improved performance in generating more comprehensive EHRs, as reflected in higher EPS values. However, this increased performance was accompanied by a notable escalation in both gender and racial biases, highlighting a performance-bias trade-off. The study identified 4 key findings as follows: (1) as model size increased, EHR generation improved, but demographic biases also became more pronounced; (2) biases were observed across all models, not just the larger ones; (3) gender bias closely aligned with real-world disease prevalence, while racial bias was evident in only a subset of diseases; and (4) racial biases varied, with some diseases showing overrepresentation of White or Black populations and underrepresentation of Hispanic and Asian groups. These findings underline the need for effective bias mitigation strategies and the development of benchmarks to ensure fairness in artificial intelligence applications for health care.

Validating Radiology Artificial Intelligence Model Performance on Photon-Counting CT Images Using Large Language Models for Ground Truth Extraction.

Objective: To compare the diagnostic accuracy of two advanced large language models (LLMs), ChatGPT-o1 and DeepSeek-V3, in expert-validated simulated otorhinolaryngology cases, and to assess subspecialty-specific performance and inter-rater agreement relative to human specialists.Methods: A cross-sectional diagnostic accuracy study was conducted using 70 expert-validated clinical vignettes across five ENT subspecialties. Two academic otolaryngologists and two LLMs independently evaluated each case. All LLMs operated in deterministic mode (temperature = 0) with standardized single-pass prompting in isolated sessions. Diagnostic accuracy, inter-rater agreement (Cohen’s κ), and subspecialty-specific performance were analyzed. A post hoc power analysis (Cohen’s h = 0.22; α = 0.05) assessed the ability to detect moderate effect sizes.Results: Both LLMs achieved a diagnostic accuracy of 90.0% (63/70), with no significant difference between them (p = 1.00) and substantial inter-model agreement (κ = 0.68). Human evaluators achieved accuracies of 97.1% and 92.9%, with fair inter-rater agreement (κ = 0.26). Subspecialty performance was highest in otology and pediatric ENT (100%) and rhinology (92.3%), with greater variability observed in laryngology and head and neck surgery. Shared error patterns included overestimation of malignancy in high-risk patients. Post hoc power analysis demonstrated 78% power to detect moderate differences.Conclusion: In controlled, vignette-based evaluations, ChatGPT-o1 and DeepSeek-V3 demonstrated diagnostic accuracy approaching expert-level performance across simulated ENT scenarios, with strong inter-model agreement and subspecialty-dependent variability. These findings highlight the potential of LLMs as diagnostic decision-support tools while underscoring the need for multimodal and real-world validation before clinical implementation.

Large language models for toxicity extraction in oncology trials: A real-world benchmark in prostate radiotherapy.

Multidisciplinary teams (MDTs) are central to treatment planning for colorectal cancer liver metastases (CRCLM) but require time and consistent access to expertise. Chat-based large language models (LLMs) such as ChatGPT can generate recommendations from written clinical summaries; however, their concordance with MDT decisions in CRCLM is not well characterized. We conducted a single-center retrospective concordance study of 30 consecutive CRCLM cases discussed at an MDT. ChatGPT was provided a standardized anonymized text synopsis (without direct imaging access) and asked for management recommendations under two a priori conditions: (1) baseline synopsis only, and (2) a conditional query in which resectability status was explicitly specified. Each case and condition was queried three independent times in separate sessions using identical prompts; outputs were mapped to predefined management categories. Agreement between the final LLM recommendation and MDT decisions was assessed using percent agreement and Cohen's kappa. Across repeated runs, the LLM assigned the same management category in all cases (within-model consistency 100%, 3/3) for both querying conditions. In the baseline condition, agreement with MDT decisions was 66.7% (20/30; Cohen's kappa = 0.606, moderate agreement). In the conditional resectability-specified condition, agreement was 93.3% (28/30; Cohen's kappa = 0.924, very good agreement). Baseline discordant cases were characterized by conservative model outputs, including recommendations for systemic therapy and/or additional diagnostic work-up; only two cases remained discordant after resectability was specified. A chat-based LLM showed moderate concordance with unanimous MDT recommendations from minimal case summaries and very good concordance when resectability status was explicitly specified. These findings support feasibility as a supervised decision-support adjunct, but do not establish clinical benefit; prospective outcome-based validation is required.

Context and Motivation The combination of Large Language Models (LLMs) and robots can be called a paradigm shift in the area of human-robot communication. The inflexible and programmed nature of traditional robotic interfaces has not allowed human and machine communication to be free and natural. With the emergence of new state of the art LLMs like GPT-4 and PaLM 2, natural language understanding, reasoning, and contextual adaptation in robotics have become possible. It is this combination that gives robots the ability to understand human complex instructions, produce meaningful replies and act to respond in accordance to the human intent- thus mediating linguistic intelligence and embodied action. Technological Overview LLMs have shown a high level of competence in semantic understanding, language translation and contextual reasoning. With robotic perception and control modules, they give a common cognitive layer, which mediates the transmission of human instruction and robotic performance. This structure enables robots to reason on the high level concerning tasks, environments and social cues. An example is that with the help of neural networks, ambiguity in a command (such as tidy the room) can be interpreted through reasoning over sensory input and sequence planning to move a robot system from a passive agent to an active partner. Human-Centered Implications Naturalistic communication is not just restricted to the linguistic interaction; it encompasses emotional indicators, sense of context and flexibility. Empathy, uncertainty negotiation, and clarifying the intent can be simulated by the use of dialogues by LLM-empowered robots, and this is greatly beneficial in building trust and usability. Use cases in medical and educational, customer service, and industrial cooperation prove that the combination of LLM-based robots is more efficient and engaging due to human-like interaction. Nevertheless, ethical transparency and avoiding overanthropomorphization is still a challenge to be maintained. Research Scope and Objectives. This paper explores how, what, and how they can be used to integrate LLMs with robotic systems in order to have naturalistic communication. It studies a multimodal fusion strategy, a grounding strategy connecting language with perception and action, as well as reinforcement learning strategy to produce continuous adaptation. The paper also discusses human factors, such as trust, interpretability and usability that determine effectiveness of communication between human beings and the robots that are controlled by the LLM. Contribution and Significance. The study provides a multifaceted approach to the integration of LLM and robots, which summarizes the existing literature and suggests the design concepts of ethical, interpretable, and context-driven dialogue between humans and robots. It features the rising trends which include embodied AI, multimodal transformers, and simulation-to-real transfer learning, which altogether characterize the next generation of communicative, intelligent, and emotionally adaptive robots.

Objectives This study assesses the feasibility, inter-rater reliability, and accuracy of using OpenAI's ChatGPT-4 and Google’s Gemini Ultra large language models (LLMs), for Emergency Medical Services (EMS) quality assurance. The implementation of these LLMs for EMS quality assurance has the potential to significantly reduce the workload on medical directors and quality assurance staff by automating aspects of the processing and review of patient care reports. This offers the potential for more efficient and accurate identification of areas requiring improvement, thereby potentially enhancing patient care outcomes. Methods Two expert human reviewers, ChatGPT GPT-4, and Gemini Ultra assessed and rated 150 consecutively sampled and anonymized prehospital records from 2 large urban EMS agencies for adherence to 2020 National Association of State EMS metrics for cardiac care. We evaluated the accuracy of scoring, inter-rater reliability, and review efficiency. The inter-rater reliability for the dichotomous outcome of each EMS metric was measured using the kappa statistic. Results Human reviewers showed high interrater reliability, with 91.2% agreement and a kappa coefficient 0.782 (0.654-0.910). ChatGPT-4 achieved substantial agreement with human reviewers in EKG documentation and aspirin administration (76.2% agreement, kappa coefficient 0.401 (0.334-0.468), but performance varied across other metrics. Gemini Ultra’s evaluation was discontinued due to poor performance. No significant differences were observed in median review times: 01:28 min (IQR 1:12 − 1:51 min) per human chart review, 01:24 min (IQR 01:09 − 01:53 min) per ChatGPT-4 chart review (p = 0.46), and 01:50 min (IQR 01:10-03:34 min) per Gemini Ultra review (p = 0.06). Conclusions Large language models demonstrate potential in supporting quality assurance by effectively and objectively extracting data elements. However, their accuracy in interpreting non-standardized and time-sensitive details remains inferior to human evaluators. Our findings suggest that current LLMs may best offer supplemental support to the human review processes, but their current value remains limited. Enhancements in LLM training and integration are recommended for improved and more reliable performance in the quality assurance processes.

Large language models (LLMs) are increasingly being applied to analyze clinical data, primarily clinical text, with an increasing emphasis on integration in health care. However, the use of LLMs in pediatric care remains underexplored. To map the emerging literature on LLM use in pediatrics involving clinical text and identify evidence gaps and future directions for implementation and evaluation. PubMed/MEDLINE, Embase, Web of Science, Scopus, and preprint servers were searched for English-language original research published from January 1, 2020, to July 1, 2025. Included studies used modern transformer-based LLMs with pediatric clinical text as input. Two reviewers independently screened studies using predefined criteria. Data were extracted by one reviewer and verified by another. Findings were descriptively synthesized, and adherence to the Minimum Information for Medical AI Reporting (MINIMAR) standards was assessed. The review included 40 studies published between 2023 and 2025. Twenty-three studies were conducted in the US, and all were retrospective observational studies using clinical data from sources such as electronic health records. Participant sample sizes ranged from 10 to 172 683. Although all pediatric age subgroups were represented, early childhood populations (aged 0-5 years) were underrepresented. The most common LLM clinical applications were diagnostic decision support in 24 studies (60.0%) and treatment planning in 7 studies (17.5%). Although all 40 studies conducted clinical evaluation of LLMs and 30 included discussions of ethics or data privacy, 39 studies (97.5%) did not meet full MINIMAR standards, 34 (85.0%) did not report use of Health Insurance Portability and Accountability Act-compliant models, and 30 (75.0%) lacked fine-tuning for pediatric-specific data. Among 33 studies assessing model performance against human annotations, 10 (30.3%) did not include clinicians as annotators; among 26 studies with multiple annotators, only 9 (34.6%) reported interannotator agreement statistics. This scoping review found that diagnostic decision support and treatment planning were commonly proposed applications of LLMs in pediatrics. However, gaps in scientific rigor and limited use of pediatric-specific data may hinder their safe and effective implementation in pediatrics. Future studies should use standardized evaluation and reporting methods, increase clinician involvement, and expand research to underrepresented ages and clinical applications.