USMLE Scores Predict Success in ABEM Initial Certification: A Multicenter Study

The numeric score for the United States Medical Licensing Examination Step 1 is one of the only universal, objective, scaled criteria for comparing the many students who apply to orthopaedic surgery residency. However, on February 12, 2020, it was announced that Step 1 would be transitioning to pass/fail scoring. The purpose of this study was to (1) determine the most important factors used for interview and resident selection after this change and (2) to assess how these factors have changed compared with a previous report on resident selection. A survey was distributed to the program directors (PDs) of all 179 orthopaedic surgery programs accredited by the Accreditation Council for Graduate Medical Education. Questions focused on current resident selection practices and the impact of the Step 1 score transition on expected future practices. A total of 78 PDs (44%) responded to the survey. Over half of PDs (59%) responded that United States Medical Licensing Examination Step 2 clinical knowledge (CK) score is the factor that will increase most in importance after Step 1 transitions to pass/fail, and 90% will encourage applicants to include their Step 2 CK score on their applications. The factors rated most important in resident selection from zero to 10 were subinternship performance (9.05), various aspects of interview performance (7.49 to 9.01), rank in medical school (7.95), letters of recommendation (7.90), and Step 2 CK score (7.27). Compared with a 2002 report, performance on manual skills testing, subinternship performance, published research, letters of recommendations, and telephone call on applicants' behalf showed notable increases in importance. As Step 2 CK is expected to become more important in the residency application process, current applicant stress on Step 1 scores may simply move to Step 2 CK scores. Performance on subinternships will remain a critical aspect of residency application, as it was viewed as the most important resident selection factor and has grown in importance compared with a previous report.

To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change. Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group. Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships. Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.

In recent years, the evaluation metrics for dermatology residency applications have changed, particularly following the transition of the United States Medical Licensing Examination (USMLE)Step 1 to a pass/fail grading system. This retrospective analysis examines dermatology residency match data from 2020 to 2024, focusing on trends in USMLE Step 2 Clinical Knowledge (CK) scores and research publications. Data extracted from the National Resident Matching Program documents revealed a notable increase in unmatched applicants with Step 2 CK scores above 250 and a decrease in matched applicants with similar scores. Concurrently, research output has risen, with matched applicants reporting over 25 publications increasing from 80 in 2020 to 121 in 2024, and a corresponding increase among unmatched applicants from 10 to 35 over the same period. Despite these improvements in individual metrics, the overall match rates for applicants with high Step 2 CK scores and extensive research portfolios have declined, suggesting that excellence in these areas alone does not guarantee a successful match.

BackgroundUnited States Medical Licensing Examination (USMLE) Step 1 is a common metric looked at by residency programs to determine invitations for candidates to interview. However, USMLE Step 2 Clinical Knowledge (CK) has also been an important factor for selecting applicants to interview and plays a significant role during applicant selection. This study aims to identify academic performance measures that correlate with USMLE Step 2 CK scores and to develop a model to predict USMLE Step 2 CK scores using previous academic measures from the first two cohorts in the longitudinal interleaved clerkship (LInC) at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM).SettingThe KSOM is a newly accredited US allopathic medical school that accepted its first class in 2017. At KSOM, a LInC model is used in the primary clinical year. In this model, rotations are two weeks in duration before moving on to the next specialty. Students complete the National Board of Medical Examiners (NBME) subject examinations in all six specialties in one week at the midpoint and the end of the LInC. Students who passed an exam at the midpoint can opt out of that exam at the end as the higher of the two exam scores is recorded. However, most students choose to take all the exams again to improve their scores and prepare for USMLE Step 2 CK.MethodologyAcademic performance measures were gathered from the class of 2021 and 2022 (n = 101) including undergraduate grade point average (GPA), undergraduate science GPA, medical college admission test score, USMLE Step 1 score, NBME clinical subject exam scores, and USMLE Step 2 CK scores. Pearson correlations were run between the performance variables and USMLE Step 2 CK scores to measure influence variables individually, then a regression model measured impacts of variables together.ResultsAll variables except undergraduate science GPA significantly correlated with USMLE Step 2 CK score. USMLE Step 1 had the strongest correlation (r = 0.752, p < 0.001). The regression model had an R of 0.859 with the internal medicine subject exam showing the highest beta coefficient (0.327, p < 0.001).ConclusionsThis study determined that USMLE Step 2 CK scores can be effectively predicted using available performance measures. With USMLE Step 1 becoming pass/fail in January 2022, the importance of USMLE Step 2 CK as a screening tool in the residency application process will likely increase. This study was conducted within a LInC curriculum and may have limited value in the prediction of scores within other clinical year curricula.

Determine whether the National Board of Medical Examiners (NBME) Subject Examination performance from six clerkships correlated with United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) Examination scores. Also, examine correlations between medical students' preclinical and clinical year mean cumulative grade point average (GPA), subject exam, and USMLE performance. The sample consisted of 507 students from the 2008-2010 graduating classes from the authors' medical school. Pearson correlations followed by stepwise linear regressions were used to investigate variance in USMLE Steps 1 and 2 CK scores explained by subject exam scores and GPA. Data from 484 (95.5%) students were included. USMLE Steps 1 and 2 CK scores had moderate-to-large positive correlations with all subject exam scores and with both GPA variables. Correlations between composite subject exam scores and USMLE Steps 1 and 2 CK exams were 0.69 and 0.77, respectively. Regression analysis demonstrated that subject exams and GPA accounted for substantial variance in Steps 1 and 2 CK exam scores (62% and 61%); when entered into the regression model first, primary care clerkship subject examination scores accounted for most of this variance. The moderate-to-large correlations between subject exam performance and USMLE scores provide reassurance that subject exam scores are associated with USMLE performance. Furthermore, the considerable variance in USMLE scores accounted for by primary care NBME scores may be due to primary care topics being reinforced through all clerkships and comprising a significant portion of the USMLE examinations, particularly Step 2 CK.

This study examines: (1) the relationships between examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) performance; (2) the effect of gender and examination timing (time per item) on the relationship between Steps 1 and 2 CK; and (3) the effect of school characteristics on the relationships between examinee characteristics and Step 2 CK performance. A series of hierarchical linear models (examinees-nested-in-schools) predicting Step 2 CK scores was fit to the data set. The sample included 54,487 examinees from 114 U.S. Liaison Committee on Medical Education-accredited medical schools. Consistent with past examinee-level research, women generally outperformed men on Step 2 CK, and examinees who received more time per item generally outperformed examinees who received less time per item. Step 1 score was generally more strongly associated with Step 2 CK performance for men and for examinees who received less time per item. School-level characteristics (size, average Step 1 performance) influenced the relationship between Steps 1 and 2 CK. Both examinee-level and school-level characteristics are important for understanding Step 2 CK performance.

To examine internal medicine residency program directors' (PDs') screening practices and perceptions about current recruitment challenges. In March-May 2017, the Association of Program Directors in Internal Medicine Survey Committee sent a survey to 373 Alliance for Academic Internal Medicine member residency programs. PDs rated the importance of 23 inclusion and 11 exclusion criteria for interview invitation decision making, provided United States Medical Licensing Examination (USMLE) cutoff scores for U.S. medical school and international medical graduates, and indicated changes in recruitment practices due to application inflation, including their ability to conduct holistic review and interest in potential solutions to address application inflation. Exploratory factor analysis was used to identify and confirm factors that were most important to interview invitation decision making. The response rate for eligible programs was 64% (233/363). USMLE Step 2 Clinical Knowledge scores were the criteria most frequently reported to be "very important" (131/233, 57%). Among respondents who reported any criteria as "very important," 155/222 (70%) identified a single most important (SMI) criterion. Non-USMLE criteria were frequently reported as an SMI criterion (68%). Concerning exclusion criteria, 157/231 (68%) reported they "absolutely would not invite" applicants with hints of unprofessional behavior. Of the 214/232 (92%) who reported an increase in applications, 138 (64%) adjusted recruitment practices. Respondents were most interested in limiting the number of applications per applicant (163/231, 71%), allowing applicants to indicate high interest in a subset of programs (151/229, 66%), and creating a national database of qualities of matched applicants for each program (121/228, 53%). PDs rely heavily on USMLE scores when making interview invitation decisions. However, collectively, non-USMLE criteria were more frequently reported as an SMI criterion. Most programs adjusted recruitment practices to respond to application volume. Several potential solutions to address application inflation garnered wide support.

This study aims to show whether correlation exists between pediatric residency applicants' quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents' ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones. In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance. Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents). These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P < .05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P < .05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P < .05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.

A 15-Year Analysis of International Medical Graduates Matching Into Diagnostic Radiology Residency Programs in the United States

Skin of color representation in medical education: An analysis of National Board of Medical Examiners' self-assessments and popular question banks

Background: Scores on “high-stakes” multiple choice exams such as the United States Medical Licensing Examination® (USMLE) are important screening and applicant ranking criteria used by residencies.Objective: We tested the hypothesis that USMLE scores do not predict overall clinical performance of emergency medicine (EM) residents.Methods: All graduates from our University-based EM residency between the years 2008 and 2015 were included. Residents who had incomplete USMLE records were terminated, transferred out of the program, or did not graduate within this timeframe were excluded from the analysis. Clinical performance was defined as a gestalt of the residency program’s leadership and was classified into three sets: top, average, and lowest clinical performer. Dissimilarities of the initial blind rankings were adjudicated during a consensus conference.Results: During the eight years of the study period, there were a total of 115 graduating residents: 73 men (63%) and 42 women. Nearly all of them (109; 95%) had allopathic medical degrees; the remainder had osteopathic degrees. There was not a statistically significant correlation between our ranking of clinical performance and the Step 2 Clinical Knowledge score. There was a non-significant correlation between clinical performance and the Step 1 score.Conclusion: Neither USMLE Step 1 nor Step 2 Clinical Knowledge were good predictors of the actual clinical performance of residents during their training. We feel that their scores are overemphasized in the resident selec­tion process.

The Learning Preferences of Applicants Who Interview for General Surgery Residency: A Multiinstitutional Study

To the Editor: On February 12, 2020, the National Board of Medical Examiners announced that United States Medical Licensing Examination (USMLE) Step 1 would be reported as a pass/fail outcome. Since the decision to no longer assign a numerical score was a surprising change, we conducted a study to analyze the online medical education community's response (known as #MedEd) to this announcement on Twitter. We used the program Tweet Archivist to capture tweets containing the hashtags #USMLE, #Step1, and/or #USMLEPASSFAIL posted during the 72-hour period surrounding the announcement. We categorized the authors; sorted tweets as irrelevant, neutral, in favor, or against the change; and tallied reasons for opinionated tweets. Following the announcement, we captured 2,056 relevant tweets (695 original, 1,361 retweets). Of these, about one-third were in favor of the change (34%), about two-thirds were neutral (59%), and a small minority were against the change (8%). Of physicians, slightly less than half tweeted in favor (42%) and neutrally toward the change (40%), while about one-fifth tweeted against it (18%). The distribution for students was similar: 39% tweeted in favor, 40% tweeted neutrally, and 21% tweeted against it. On average, tweets in favor were retweeted significantly more than those in opposition (1.5 vs 0.2 times). The majority of original tweets were sent by physicians (57%), followed by students (26%) and users who could not be classified (18%). The 3 most common reasons tweeted by those in favor of the change were as follows: (1) clinical knowledge will be improved as students will no longer focus heavily on Step 1, (2) Step 1 is a poor test that was never designed to rank applicants, and (3) this will foster holistic review. Conversely, the 3 most common concerns tweeted by those against the change were the following: (1) the importance of a Step 1 score will be replaced by another metric, such as the Step 2 Clinical Knowledge score, (2) this change will hurt international medical graduates, and (3) a standardization tool is necessary to compare applicants. These concerns are not without merit as they are the same top 3 concerns shared by program directors in a recent study by Makhoul and colleagues.1 Overall, the Step 1 score change announcement was well received on #MedEd Twitter. More users tweeted in favor of the change than against it, and positive tweets were retweeted more often than negative ones.

BackgroundCombined Internal Medicine/Pediatrics (Med/Peds) residencies rely on categorical program data to predict pass rates for the American Board of Internal Medicine Certifying Exam (ABIM-CE) and the American Board of Pediatrics Certifying Exam (ABP-CE). There is insufficient literature describing what best predicts a Med/Peds resident passing board exams. In this study, we aimed to determine how standardized test scores predict performance on ABIM-CE and ABP-CE for Med/Peds residents.MethodologyWe analyzed prior exam scores for 91/96 (95%) residents in a Med/Peds program from 2008 to 2017. Scores from the United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) and In-Training Exams in Internal Medicine (ITE-IM) and Pediatrics (ITE-P) were analyzed with the corresponding ABIM-CE and ABP-CE first-time scores. Linear and logistic regression were applied to predict board scores/passage.ResultsUSMLE 1 and 2 CK, ITE-IM, and ITE-P scores had a linear relationship with both ABIM-CE and ABP-CE scores. In the linear regression, adjusted R2 values showed low-to-moderate predictive ability (R2 = 0.11-0.35), with the highest predictor of ABIM-CE and ABP-CE being USMLE Step 1 (0.35) and Postgraduate Year 1 (PGY-1) ITE-IM (0.33), respectively. Logistic regression showed odds ratios of passing board certifications ranging from 1.05 to 1.53 per point increase on the prior exam score. The PGY-3 ITE-IM was the best predictor of passing both certifying exams.ConclusionsIn one Med/Peds program, USMLE Steps 1 and 2 and all ITE-IM and ITE-P scores predicted certifying exam scores and passage. This provides Med/Peds-specific data to allow individualized resident counseling and guide programmatic improvements targeted to board performance.