Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation

Background: Generative Artificial Intelligence (AI), particularly large language models like ChatGPT, has rapidly entered medical education. In pharmacology instruction within MBBS programs, these AI tools offer novel educational benefitsfrom ondemand tutoring to personalized learningbut also pose risks due to "hallucinations, " i. e., AI -generated misinformation. Objective: This review explores how generative AI is being integrated into pharmacology education, the educational advantages it provides, the nature and causes of AI hallucinations, and the potential risks of relying on inaccurate AI -generated content in medical learning. Mitigation strategies, best practices, case examples, ethical considerations, and future directions are discussed to inform balanced and safe use. Methods: A literature review was conducted focusing on recent (2020-2025) peer -reviewed sources and authoritative reports on generative AI in medical and pharmacology education. Results: Generative AI can enhance pharmacology education through adaptive learning, rapid information access, and interactive simulations, which students have found useful for studying and clinical preparation. However, AI hallucinationsplausible -sounding yet incorrect answersare common, with studies reporting a substantial fraction of AI -generated medical responses containing errors. These inaccuracies can mislead learners and threaten patient safety if not recognized. Strategies such as careful validation of AI -provided information, AI literacy training, and technical approaches (e. g. retrieval -augmented generation) are identified to mitigate risks. Conclusions: Generative AI holds significant promise in medical pharmacology education by augmenting learning and engagement. Yet, educators and students must remain vigilant about AI hallucinations. A balanced approachleveraging AI's benefits while implementing safeguards and ethical guidelinesis essential to ensure that AI becomes a reliable adjunct in pharmacology education rather than a source of misinformation. Ongoing research, policy development, and training are needed to fully realize generative AI's potential in medical education while minimizing its risks.

BackgroundClinical reasoning is an essential medical competence that should be taught and assessed from the beginning of medical studies. These skills can be evaluated using the Script Concordance Test (SCT), which presents daily clinical scenarios characterised by uncertainty. Due to the lack of validated research on this method in Germany, particularly in the field of neurology, we developed and implemented an SCT at Ruhr University Bochum. We compared different teaching methods (clinical seminar vs. digital video course) and their outcomes on the examination.MethodsA group of 6th-year medical students who had received the same education completed an SCT after participating in either a clinical seminar or a digital video course. The SCT was developed using blueprints on stroke and epilepsy. The test consisted of 40 case vignettes with a total of 120 items. Initially, experts completed the test to establish the reference panel. The final high-stakes examination was created using the aggregate scoring method and an item analysis.ResultsThe SCT was completed by 15 experts and 59 students. The final SCT consisted of 112 items and achieved a Cronbach’s alpha of 0.85. A significant difference (p < 0.05) was observed between the experts, who achieved a mean score of 81.75, and the students on the first assessment day, who achieved a mean score of 68.92. No significant differences were found between the groups (interactive video course and seminar) or assessment time points. The questionnaire revealed a sense of insecurity in clinical decision-making before the SCT and highlighted the need to incorporate clinical reasoning practices from the beginning of medical studies to mitigate fear in uncertain situations. The SCT helped students structure decision-making processes and and improved their confidence in making decisions.ConclusionThe SCT is a reliable and valid tool for assessing medical students throughout their university education. Regular exposure to the SCT format would facilitate familiarity with its structure. We propose utilising the SCT as a learning tool rather than solely for assessment purposes. For instance, it could be integrated into teaching methodologies as a think-aloud exercise or incorporated into progress tests.

Clinical reasoning is a complex skill that represents a trainee's ability to use their professional knowledge and skills to assess and solve the problems that arise in clinical practice. As an integral tenet of the genetic counseling process, clinical reasoning skills underlie many of the Practice-Based Competencies (2019) across a variety of domains. Despite the long-lasting recognition of the importance of this complex skill in the training of genetic counselors, clinical reasoning has traditionally been difficult to assess in a standardized way in healthcare education. Script concordance testing is a standardized method of assessing clinical reasoning skills in ambiguous clinical situations. The tool has been used to successfully measure the clinical reasoning skills of trainees in various healthcare training programs and has never been used in a genetic counseling training program. We conducted a pilot study to assess the utility of script concordance testing in the field of genetic counseling as an objective measure of clinical reasoning in trainees. The script concordance test was constructed for the field of genetic counseling and administered to 22 second year genetic counseling students in the Joan H. Marks Graduate Program in Human Genetics at Sarah Lawrence College. Twelve genetic counselors served on a panel to provide expert clinical reasoning responses and a scoring grid was developed using the aggregate scores method. The utility of the tool was measured using Cronbach's alpha coefficient, and scores of students and the panel were compared using Hedge's g. Results revealed statistically significant differences between the scores of panelists and students and good reliability. This study shows that script concordance testing can be used to measure clinical reasoning skills in genetic counseling trainees in a way that is reliable, standardized, and easy to use, thereby allowing programs to better assess the clinical reasoning skills of trainees prior to graduation.

Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high-stakes testing. This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats. We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice. Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter-panel, inter-panellist and test-retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct-irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct-irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended. Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

BackgroundIn recent years, generative artificial intelligence and large language models (LLMs) have rapidly advanced, offering significant potential to transform medical education. Several studies have evaluated the performance of chatbots on multiple-choice medical examinations.ObjectiveThe study aims to assess the performance of two LLMs—GPT-4o and OpenAI o1—on the Médico Interno Residente (MIR) 2024 examination, the Spanish national medical test that determines eligibility for competitive medical specialist training positions.MethodsA total of 176 questions from the MIR 2024 examination were analyzed. Each question was presented individually to the chatbots to ensure independence and prevent memory retention bias. No additional prompts were introduced to minimize potential bias. For each LLM, response consistency under verification prompting was assessed by systematically asking, “Are you sure?” after each response. Accuracy was defined as the percentage of correct responses compared to the official answers provided by the Spanish Ministry of Health. It was assessed for GPT-4o, OpenAI o1, and, as a benchmark, for a consensus of medical specialists and for the average MIR candidate. Subanalyses included performance across different medical subjects, question difficulty (quintiles based on the percentage of examinees correctly answering each question), and question types (clinical cases vs theoretical questions; positive vs negative questions).ResultsOverall accuracy was 89.8% (158/176) for GPT-4o and 90% (160/176) after verification prompting, 92.6% (163/176) for OpenAI o1 and 93.2% (164/176) after verification prompting, 94.3% (166/176) for the consensus of medical specialists, and 56.6% (100/176) for the average MIR candidate. Both LLMs and the consensus of medical specialists outperformed the average MIR candidate across all 20 medical subjects analyzed, with ≥80% LLMs’ accuracy in most domains. A performance gradient was observed: LLMs’ accuracy gradually declined as question difficulty increased. Slightly higher accuracy was observed for clinical cases compared to theoretical questions, as well as for positive questions compared to negative ones. Both models demonstrated high response consistency, with near-perfect agreement between initial responses and those after the verification prompting.ConclusionsThese findings highlight the excellent performance of GPT-4o and OpenAI o1 on the MIR 2024 examination, demonstrating consistent accuracy across medical subjects and question types. The integration of LLMs into medical education presents promising opportunities and is likely to reshape how students prepare for licensing examinations and change our understanding of medical education. Further research should explore how the wording, language, prompting techniques, and image-based questions can influence LLMs’ accuracy, as well as evaluate the performance of emerging artificial intelligence models in similar assessments.

Residents in tutored practice exchange groups have better medical reasoning as measured by script concordance test: a controlled, nonrandomized study

As generative artificial intelligence (AI) tools and large language models (LLMs)-powered applications develop rapidly in the era of algorithms, it should be integrated thoughtfully to enhance English as a Foreign Language (EFL) teaching and learning without replacing learners’ critical thinking (CT). This study systematically analyzes the impact of generative AI tools and LLMs on language learners’ CT in EFL education using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework to identify, evaluate, and synthesize relevant studies from 2022 to 2025. A thorough review of 15 selected studies focuses on generative AI tools and LLMs’ dual nature, research methods, main focuses, theory and models, limitations and challenges, and future directions in the field based on Web of Science (WoS), SCOPUS, ERIC, ProQuest, and Google Scholar. The findings identified generative AI tools and LLMs possessed both the potential to nurture and the risk of hindering CT in EFL education. 66.67% of studies reported generative AI tools and LLMs’ positive role in CT, while 33.33% of studies reported its negative role in CT. Furthermore, 3 types of research methods, 3 key themes of research focus, and 4 groups of theoretical perspectives were examined. However, 4 kinds of limitations in this field remain, including research scope, user dependency, generative AI reliability, and pedagogical integration. Future research can focus on assessing long-term effects, broadening research scope, promoting responsible AI use, and refining pedagogical strategies. Finally, Limitations, implications and future direction of this study were discussed.

The script concordance test (SCT) is used in health professions education to assess a specific facet of clinical reasoning competence: the ability to interpret medical information under conditions of uncertainty. Grounded in established theoretical models of knowledge organization and clinical reasoning, the SCT has three key design features: (1) respondents are faced with ill-defined clinical situations and must choose between several realistic options; (2) the response format reflects the way information is processed in challenging problem-solving situations; and (3) scoring takes into account the variability of responses of experts to clinical situations. SCT scores are meant to reflect how closely respondents’ ability to interpret clinical data compares with that of experienced clinicians in a given knowledge domain. A substantial body of research supports the SCT's construct validity, reliability, and feasibility across a variety of health science disciplines, and across the spectrum of health professions education from pre-clinical training to continuing professional development. In practice, its performance as an assessment tool depends on careful item development and diligent panel selection. This guide, intended as a primer for the uninitiated in SCT, will cover the basic tenets, theoretical underpinnings, and construction principles governing script concordance testing.

Pediatric Hospitalists' Performance and Perceptions of Script Concordance Testing for Self-Assessment.

In today’s rapidly evolving business landscape, Artificial Intelligence (AI), and specifically Large Language Models (LLMs), are redefining how organizations operate, make decisions, and engage with customers. AI-driven technologies have become indispensable, providing businesses with powerful tools to streamline operations, derive actionable insights from vast data, and foster more meaningful customer interactions. For business leaders, scholars, and practitioners alike, understanding the transformative potential of AI isn’t just advantageous—it’s essential to staying competitive in an increasingly data-driven world.This editorial delves into recent scholarly advancements in LLM applications within business contexts, analyzing studies that explore AI’s potential across various domains, from decision support to creative industries. By introducing a structured framework, this editorial highlights key insights and contributions from recent studies, assessing their value to academia and industry. The following comparative analysis sheds light on how these innovations shape our understanding of AI’s role in business while pointing to future research directions.Puyt and Madsen's (2024) study stands out as a foundational exploration of LLM accuracy, assessing ChatGPT-4's ability to recount the history of the SWOT analysis-a vital business strategy tool. Their findings reveal that, while ChatGPT-4 effectively conveys general concepts, it struggles with detailed historical information, often producing inaccuracies or "hallucinations." This gap underscores the need for LLMs to be trained with verified academic data, particularly for strategic business applications that demand precision. This study not only contributes to the literature by proposing methods to evaluate AI accuracy in historical contexts but also highlights the importance of rigorous information vetting in industry settings where reliability is crucial.In contrast, Raikov et al. (2024) explore a hybrid intelligence model that combines LLM capabilities with explainable AI (XAI) principles to enhance human-machine collaboration. Their approach emphasizes cognitive semantics, improving transparency and decision-making efficiency. The hybrid model's real-time adaptability addresses the needs of complex, regulated industries such as finance and healthcare, where trust in AI decisions is paramount. Academically, this study provides a valuable addition to XAI literature by demonstrating how LLMs can bridge the gap between AI autonomy and human oversight, making it a model for future human-AI interactions in complex business environments.Another significant study by Mariotti and colleagues (2024) examines the integration of LLMs with enterprise knowledge graphs to enhance data-driven decision-making. By enabling organizations to leverage knowledge graphs for more accurate and scalable data retrieval, this research provides a robust framework for businesses seeking efficient knowledge management systems. The academic contribution here lies in advancing the dialogue between LLMs and knowledge graphs, emphasizing ethical data handling and quality standards essential for industry applications. For enterprises, the study offers practical solutions to achieve streamlined data management, balancing automation with privacy and security. 2024) take a different approach, investigating LLMs' role in creative industries, specifically within fashion design. They introduce a hybrid intelligence model that supports creative processes, allowing AI to complement rather than replace human ingenuity. While LLMs in this field demonstrate potential in automating repetitive design tasks and enhancing customer personalization, the study reveals limitations in AI's ability to handle spatial and stylistic nuances. This study's academic contribution lies in promoting human-AI co-creation, inspiring further research into AI applications across diverse creative sectors, including media and marketing.Collectively, these studies not only illuminate LLMs' transformative potential in business but also highlight critical ethical and operational considerations. Ensuring accuracy, transparency, and data privacy are vital to responsibly integrating AI into business workflows. Future research should focus on enhancing LLM accuracy, refining hybrid intelligence models, and exploring creative AI applications, all while maintaining ethical standards. As LLMs evolve, interdisciplinary collaborations will be essential to harness their full potential, making AI an ethical, effective, and innovative force in the business world.

Development and validation of a script concordance test to assess biosciences clinical reasoning skills: A cross-sectional study of 1st year undergraduate nursing students

Programmes of assessment should measure the various components of clinical competence. Clinical reasoning has been traditionally assessed using written tests and performance-based tests. The script concordance test (SCT) was developed to assess clinical data interpretation skills. A recent review of the literature examined the validity argument concerning the SCT. Our aim was to provide potential users with evidence-based recommendations on how to construct and implement an SCT. A systematic review of relevant databases (MEDLINE, ERIC [Education Resources Information Centre], PsycINFO, the Research and Development Resource Base [RDRB, University of Toronto]) and Google Scholar, medical education journals and conference proceedings was conducted for references in English or French. It was supplemented by ancestry searching and by additional references provided by experts. The search yielded 848 references, of which 80 were analysed. Studies suggest that tests with around 100 items (25-30 cases), of which 25% are discarded after item analysis, should provide reliable scores. Panels with 10-20 members are needed to reach adequate precision in terms of estimated reliability. Panellists' responses can be analysed by checking for moderate variability among responses. Studies of alternative scoring methods are inconclusive, but the traditional scoring method is satisfactory. There is little evidence on how best to determine a pass/fail threshold for high-stakes examinations. Our literature search was broad and included references from medical education journals not indexed in the usual databases, conference abstracts and dissertations. There is good evidence on how to construct and implement an SCT for formative purposes or medium-stakes course evaluations. Further avenues for research include examining the impact of various aspects of SCT construction and implementation on issues such as educational impact, correlations with other assessments, and validity of pass/fail decisions, particularly for high-stakes examinations.

The Ministry of Food and Drug Safety (MFDS) of the Republic of Korea, similar to the United States Food and Drug Administration and the United Kingdom's Medicines and Healthcare products Regulatory Agency, issued specific regulatory guidelines on January 24, 2025, for approving generative artificial intelligence (AI) technologies as medical devices [1].Although these guidelines use the term 'generative AI,' they predominantly focus on the approval of AI software tools based on large language models (LLMs) and large multimodal models (LMMs) [2], the latter of which can process various types of input data, such as texts, images, videos, audio, and bio-signals.Unlike the broader guidelines for approving AI models as medical devices, specific regulatory guidelines for LLMs/LMMs have arguably not yet been proposed in other countries.

Clinical judgment is the ability to weigh clinical information and make decisions under conditions of uncertainty. Although neurological localization (NL) and neurological emergencies (NE) present such uncertainties, no validated method is reported to assess these decision-making skills. A script concordance test (SCT) was designed and validated to assess clinical judgment in NL and NE. Our SCT comprised 14 clinical scenarios (53 questions). Candidates picked the response they considered the best for the questions in each scenario. Undergraduates and internal medicine residents completed the SCT; their responses were scored against the scoring key derived from an expert panel of accredited neurologists. Scores were expressed as a percentage of the maximum score. Mean total scores for undergraduates (n = 52), residents (n = 37) and experts (n = 15) were 61.0±0.9, 68.3±1.1 and 76.6±1.1 (mean±standard error of the mean, P < 0.001). Mean scores for undergraduates, residents and experts were 59.3±1.1, 66.4±1.4 and 76.1±1.8 (P < 0.001) for NL, and 62.9±1.3, 70.4±1.3 and 77.2±1.6 (P < 0.001) for NE. Senior residents scored higher than junior residents (postgraduate years 2-5 versus postgraduate year 1, 69.7±1.4 vs. 65.3±1.1, P = 0.035). The higher scores with increasing clinical experience supports the construct validity of the SCT. The SCT showed acceptable reliability (G coefficient 0.74±0.05). Our SCT is validated to reliably assess NL and NE in undergraduate and postgraduate learners; it is generalizable and feasible. It has potential as a valuable adjunct assessment tool for clinical judgment. Future plans to design SCTs to evaluate other topics in clinical neurology, as a multi-center study, are under way.

Le test de concordance de script : un outil pédagogique multimodal