National Board of Medical Examiners

Abstract Howard Wainer, Irina Grabovsky and Daniel H. Robinson explain why we should treat dramatic educational success stories in the USA with extreme caution

One concern with AI systems is their potential to produce biased output. These biases can be difficult to detect due to the complex and often proprietary nature of the systems, which limits transparency. We propose an evaluation framework for assessing whether a system exhibits biased behavior. This evaluation consists of a series of tasks in which an AI system is instructed to select one of two students for an award based on their performance on a standardized assessment. The two students are implicitly associated with different demographic subgroups, and the evaluation is designed such that students from each subgroup perform equally well on average. In this way, any consistent preference for a particular subgroup can be attributed to bias in the system’s output. The proposed framework is illustrated using GPT-3.5 and GPT-4, with racial subgroups (Black/White) and an assessment composed of math items. In this demonstration, GPT-3.5 favored Black students over White students by a factor of approximately 2:1 (66.5%; 1,061 out of 1,596 non-equivocal choices). In contrast, GPT-4 showed a slight numerical preference for Black students (291 vs. 276; 51.3%), but this difference was not statistically significant (p = 0.557), indicating no detectable bias. These results suggest that the proposed audit is sensitive to differences in system bias in this context.

Public health leadership plays a crucial role in shaping effective health policies and practices. The National Board of Public Health Examiners (NBPHE) conducts a job task analysis (JTA) survey every 5-7 years to update the Certified in Public Health (CPH) examination. The objective of this study is to examine the JTA findings on leadership tasks in public health practice. In April 2022, through the collaboration of expert panels and a validation survey, 103 tasks organized into ten domains were established for the JTA survey. The JTA survey was distributed online to current public health professionals. Across the tasks in the ten domains, respondents were asked about frequency (Scale of 1-6; how often they performed this task) and criticality (Scale of 1-5; how important this task was to their job). A total of 2,091 public health professionals responded to at least 82 of the 103 tasks (80%) and were included in the analysis. Approximately 86% of respondents worked in the United States and 41% had earned their CPH credential. Average frequency ratings ranged from 2.38 to 5.58, indicating that task ratings ranged from being performed never performed, every few years to daily. Average criticality ratings ranged from 2.46 to 4.64, indicating that task ratings ranged from not important to critically important. Specific to leadership, it was found that the 'leadership' domain ranked 2nd highest for both frequency and criticality. Our findings suggest that leadership-focused development as part of academic public health programs and continuing education for the workforce is essential. Future research may examine how individuals perform on the leadership domain of the CPH exam across multiple characteristics to better inform additional workforce development strategies.

Clinically based performance assessments, including those employing standardized patients (SPs), have been utilized to educate and assess medical students and graduates. Starting in the 1990s, demonstrating competency on these types of assessments became part of the medical licensure pathway in some countries, and was re-added to the licensure pathway in the UnitedStates in 2004. Studies have been published to support the validity of the scores and associated pass/fail decisions obtained from SP assessments. With the onset of the COVID-19 pandemic, testing organizations were forced to suspend many of their in-person clinical skills assessments. There is widespread recognition that clinical skills are necessary for high-quality patient care. Nevertheless, there has been disparate interest in reestablishing national assessments of these skills. Based on the recommendations of the Special Commission on Osteopathic Medical Licensure Assessment, the National Board of Osteopathic Medical Examiners (NBOME) established the Core Competency Capstone for Doctors of Osteopathic Medicine (C3DO). The objectives of this study were to describe the assessment and to ascertain the feasibility of administering it at multiple colleges of osteopathic medicine (COMs). Surveys of osteopathic medical school personnel and students taking the assessment were conducted to gather data on the feasibility of administering the C3DO, the challenges associated with a distributed model, and the educational value of the assessment. Reports submitted by the medical schools provided information on the cost of the assessment. Some psychometrics analyses of Phase 1 C3DO pilot data were conducted, including the quantification of measurement errors. The NBOME, in collaboration of the COMs, was able to administer the C3DO to almost the entire class at four osteopathic medical schools. Although there were challenges, including data transfer from the schools, summary scores were derived for a total of 811 students, including nine repeat test takers. Medical students across sites indicated that the C3DO was a valuable educational assessment activity. The reliability of the ratings derived from the checklists (history building [HB], physical examination [PE]) and Communication and Respectfulness Evaluation (CARE) tool were adequate. Based on a variance components analysis, there was some evidence to suggest that at a given school, the choice of an SP portraying and scoring the case could have an impact on the student scores. The C3DO-distributed site model, which includes centrally developed cases, standardized training of SPs, and independent scoring of osteopathic manipulative treatment (OMT), provides the means to assess the clinical skills of osteoapthic medical students at different testing sites. The implementation of a distributed, school-based model for the assessment of osteopathic clinical skills is feasible, potentially economically advantageous, and if rigorously developed and administered, should provide for comparable student scores and associated competency decisions. Additional Phase 2 (2024-2025) and Phase 3 (2025-2026) studies are underway to establish the psychometric rigor of the assessment more fully.

Methods for detecting item parameter drift may be inadequate when every exposed item is at risk for drift. To address this scenario, a strategy for detecting item parameter drift is proposed that uses only unexposed items deployed in a stratified random method within an experimental design. The proposed method is illustrated by investigating unexpected score increases on a high-stakes licensure exam. Results for this example were suggestive of item parameter drift but not significant at the .05 level.

Maintaining consistent item difficulty across test forms is crucial for accurately and fairly classifying examinees into pass or fail categories. This article presents a practical procedure for classifying items based on difficulty levels using functional data analysis (FDA). Methodologically, we clustered item characteristic curves (ICCs) into difficulty groups by analyzing their functional principal components (FPCs) and then employed a neural network to predict difficulty for ICCs. Given the degree of similarity between many ICCs, categorizing items by difficulty can be challenging. The strength of this method lies in its ability to provide an empirical and consistent process for item classification, as opposed to relying solely on visual inspection. The findings reveal that most discrepancies between visual classification and FDA results differed by only one adjacent difficulty level. Approximately 67% of these discrepancies involved items in the medium to hard range being categorized into higher difficulty levels by FDA, while the remaining third involved very easy to easy items being classified into lower levels. The neural network, trained on these data, achieved an accuracy of 79.6%, with misclassifications also differing by only one adjacent difficulty level compared to FDA clustering. The method demonstrates an efficient and practical procedure for classifying test items, especially beneficial in testing programs where smaller volumes of examinees tested at various times throughout the year.

Competency-based medical education (CBME) has produced large collections of data, which can provide valuable information about trainees and medical education systems. Many organizations continue to struggle with accessing, collecting, governing, analyzing, and visualizing their clinical and/or educational data. This hinders data sharing efforts within and across organizations, which are foundational in supporting system-wide improvements. Challenges to data sharing within medical education include variability in legislation, existing data policies, heterogeneity of data, inadequate data infrastructure, and various intended purposes or uses. In this eye opener, the authors describe four case studies to illustrate some of the aforementioned challenges and characterize the complexity of data sharing within medical education along two dimensions: organizational (single vs. multiple) and data type (clinical and/or educational). With the goal of better supporting data sharing initiatives, the authors introduce an action-oriented blueprint that includes a three-stage process (i.e., preparation, execution, and iteration) to highlight crucial aspects of data sharing. This evidence-informed model incorporates current best practices and aims to support data sharing initiatives within their own organizations and across multiple organizations. Finally, organizations can use this model to conceptually guide and track their progression throughout the data sharing process.

The National Board of Medical Examiners® Comprehensive Basic Science Self-Assessment (CBSSA) is a 200-item web-administered multiple-choice examination built to the same test specifications in content and statistics as the United States Medical Licensing Examination® Step 1 to help students assess their readiness and familiarize themselves with the content and format of Step 1. This study investigated the relationship between last-attempt CBSSA scores and first-attempt Step 1 pass-fail outcomes based on a large and recent sample of 23,271 US and Canadian medical students who took CBSSA under standard timing conditions between February 23, 2022 and March 1, 2023 and subsequently took Step 1 for the first time under standard timing conditions. The number of CBSSA attempts, when students took CBSSA in relation to their first Step 1 attempt, and the interaction between the different levels of these two factors were investigated. Separate binary logistic regressions were performed with last-attempt CBSSA score as the independent variable and first-attempt Step 1 pass-fail outcome as the dependent variable for the entire study group as well as sub-groups who took Step 1 within 1-2 weeks, 3-4 weeks, 5-6 weeks, and over 6 weeks of their last CBSSA attempt. All models were statistically significant. The relationship between the last CBSSA and first Step 1 was statistically different across the different time intervals, except for those within 5-6 weeks and over 6 weeks. The highest classification accuracy was for students testing within 1-2 weeks. Results suggest that CBSSA remains an effective tool to gauge readiness for Step 1.

This study investigated the relationship among school engagement, students’ emotions, and academic performance of students in grades 3-6 in South Korea. A random sampling approach was used to extract data from 1,075 students out of a total of 141,926 students who used the educational learning platform, I-TokTok, adapted as the primary Learning Management System (LMS) at the provincial level. The present study aimed to identify dimensions of school engagement by exploring the behaviors consistent with IMS Caliper Analytics Specifications, a common standard utilized for collecting learning data from digital resources. Exploratory and Confirmatory Factor Analyses revealed a three-factor model of school engagement among the 13 learning behavioral indicators: behavioral engagement, social engagement, and cognitive engagement. Students’ emotions were measured through voluntary daily activities in the platform involving reflecting on, recognizing, and recording of their emotions. Students’ academic performance was assessed with performance in math tests administered within the platform. Consistent with current literature, results demonstrated that dimensions of school engagement (i.e., behavioral and social engagement) and students’ emotions positively predicted their math performance. Lastly, school engagement mediated the relationship between students’ emotions and math performance. The present study emphasizes the importance of investigating the underlying mechanisms through which elementary students emotions and school engagement predict academic achievement in an online learning environment. This relatively new area of educational research deserves attention in the field of learning analytics. We highlight the importance of considering ways to improve both students’ emotions and their school engagement to maximize the student learning outcomes.