Can We Increase the Value and Decrease the Cost of Clinical Skills Assessment?

COMLEX-USA and USMLE for Osteopathic Medical Students: Should We Duplicate, Divide, or Unify?

IntroductionThe reliance on the United States Medical Licensing Examination (USMLE) Step 1 scores in residency selection creates problems for osteopathic medical students and the programs that review their applications. Although many osteopathic students take the USMLE to improve their standing for residency selection, students who score poorly may harm their candidacy. Simultaneously, programs unfamiliar with the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) may struggle to evaluate applicants who have not taken USMLE.ObjectiveTo determine the association between COMLEX-USA Level 1 and USMLE Step 1 scores and derive an equation that could be used to predict USMLE performance or approximate USMLE scores for applicants who have only taken COMLEX-USA.MethodsWe reviewed COMLEX-USA Level 1 and USMLE Step 1 scores for all students at the Lake Erie College of Osteopathic Medicine (LECOM), Bradenton campus, from January 2012 until December 2016. Linear regression was used to evaluate the relationship between COMLEX-USA Level 1 and USMLE Step 1 scores.ResultsOverall, 2097 students took both examinations during the study period. Every one-point increase in COMLEX-USA was associated with a 0.15 point increase in USMLE Step 1 score (standard error 11.5; model R2 0.56). On average, students scored 30 percentile points lower on USMLE Step 1 than on COMLEX-USA, and 24% of students scoring <500 on COMLEX-USA Level 1 failed USMLE Step 1.ConclusionsStudents or programs interested in predicting performance on USMLE Step 1 from performance on COMLEX-USA Level 1 can do so with the following equation: USMLE Step 1 = 0.15 (COMLEX-USA Level 1) + 138.7.

So You Want to Become a Plastic Surgeon? What You Need to Do and Know to Get into a Plastic Surgery Residency

Determine whether the National Board of Medical Examiners (NBME) Subject Examination performance from six clerkships correlated with United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) Examination scores. Also, examine correlations between medical students' preclinical and clinical year mean cumulative grade point average (GPA), subject exam, and USMLE performance. The sample consisted of 507 students from the 2008-2010 graduating classes from the authors' medical school. Pearson correlations followed by stepwise linear regressions were used to investigate variance in USMLE Steps 1 and 2 CK scores explained by subject exam scores and GPA. Data from 484 (95.5%) students were included. USMLE Steps 1 and 2 CK scores had moderate-to-large positive correlations with all subject exam scores and with both GPA variables. Correlations between composite subject exam scores and USMLE Steps 1 and 2 CK exams were 0.69 and 0.77, respectively. Regression analysis demonstrated that subject exams and GPA accounted for substantial variance in Steps 1 and 2 CK exam scores (62% and 61%); when entered into the regression model first, primary care clerkship subject examination scores accounted for most of this variance. The moderate-to-large correlations between subject exam performance and USMLE scores provide reassurance that subject exam scores are associated with USMLE performance. Furthermore, the considerable variance in USMLE scores accounted for by primary care NBME scores may be due to primary care topics being reinforced through all clerkships and comprising a significant portion of the USMLE examinations, particularly Step 2 CK.

Assessment Briefs

To determine the psychometric characteristics of diagnostic justification scores based on the patient note format of the United States Medical Licensing Examination Step 2 Clinical Skills exam, which requires students to document history and physical findings, differential diagnoses, diagnostic justification, and plan for immediate workup. End-of-third-year medical students at one institution wrote notes for five standardized patient cases in May 2013 (n = 180) and 2014 (n = 177). Each case was scored using a four-point rubric to rate each of the four note components. Descriptive statistics and item analyses were computed and a generalizability study done. Across cases, 10% to 48% provided no diagnostic justification or had several missing or incorrect links between history and physical findings and diagnoses. The average intercase correlation for justification scores ranged from 0.06 to 0.16; internal consistency reliability of justification scores (coefficient alpha across cases) was 0.38. Overall, justification scores had the highest mean item discrimination across cases. The generalizability study showed that person-case interaction (12%) and task-case interaction (13%) had the largest variance components, indicating substantial case specificity. The diagnostic justification task provides unique information about student achievement and curricular gaps. Students struggled to correctly justify their diagnoses; performance was highly case specific. Diagnostic justification was the most discriminating element of the patient note and had the greatest variability in student performance across cases. The curriculum should provide a wide range of clinical cases and emphasize recognition and interpretation of clinically discriminating findings to promote the development of clinical reasoning skills.

Medical schools using formal undergrad-uate selectivity measures do so to compensate for the psychometricinadequacies of college grade-point averages, believing that moremeaning can be derived from the GPA if it is attached to a measureof institutional performance (academic rigor) or selectivity (strin-gent admission standards).Researchers have reported mixed results on whether formal mea-sures of undergraduate institution selectivity are useful contributorsto predicting medical student performance.

Training Standardized Patients for a High-Stakes Clinical Performance Examination in the California Consortium for the Assessment of Clinical Competence

The purpose of the present study was to assess the fit of three factor analytic (FA) models with a representative set of United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) cases and examinees based on substantive considerations. Checklist, patient note, communication and interpersonal skills, as well as spoken English proficiency data were collected from 387 examinees on a set of four USMLE Step 2 CS cases. The fit of skills-based, case-based, and hybrid models was assessed. Findings show that a skills-based model best accounted for performance on the set of four CS cases. Results of this study provide evidence to support the structural aspect of validity. The proficiency set used by examinees when performing on the Step 2 CS cases is consistent with the scoring rubric employed and the blueprint used in form assembly. These findings will be discussed in light of past research in this area.

The unexpected discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam in January 2021 carries both risks and opportunities for medical education in the United States. Step 2 CS had far-reaching effects on medical school curricula and school-based clinical skills assessments. Absent the need to prepare students for this high-stakes exam, will the rigor of foundational clinical skills instruction and assessment remain a priority at medical schools? In this article, the authors consider the potential losses and gains from the elimination of Step 2 CS and explore opportunities to expand local summative assessments beyond the narrow bounds of Step 2 CS. The responsibility for implementing a rigorous and credible summative assessment of clinical skills that are critical for patient safety as medical students transition to residency now lies squarely with medical schools. Robust human simulation (standardized patient) programs, including regional and virtual simulation consortia, can provide infrastructure and expertise for innovative and creative local assessments to meet this need. Novel applications of human simulation and traditional formative assessment methods, such as workplace-based assessments and virtual patients, can contribute to defensible summative decisions about medical students' clinical skills. The need to establish validity evidence for decisions based on these novel assessment methods comprises a timely and relevant focus for medical education research.

BackgroundUnited States Medical Licensing Examination (USMLE) Step 1 is a common metric looked at by residency programs to determine invitations for candidates to interview. However, USMLE Step 2 Clinical Knowledge (CK) has also been an important factor for selecting applicants to interview and plays a significant role during applicant selection. This study aims to identify academic performance measures that correlate with USMLE Step 2 CK scores and to develop a model to predict USMLE Step 2 CK scores using previous academic measures from the first two cohorts in the longitudinal interleaved clerkship (LInC) at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM).SettingThe KSOM is a newly accredited US allopathic medical school that accepted its first class in 2017. At KSOM, a LInC model is used in the primary clinical year. In this model, rotations are two weeks in duration before moving on to the next specialty. Students complete the National Board of Medical Examiners (NBME) subject examinations in all six specialties in one week at the midpoint and the end of the LInC. Students who passed an exam at the midpoint can opt out of that exam at the end as the higher of the two exam scores is recorded. However, most students choose to take all the exams again to improve their scores and prepare for USMLE Step 2 CK.MethodologyAcademic performance measures were gathered from the class of 2021 and 2022 (n = 101) including undergraduate grade point average (GPA), undergraduate science GPA, medical college admission test score, USMLE Step 1 score, NBME clinical subject exam scores, and USMLE Step 2 CK scores. Pearson correlations were run between the performance variables and USMLE Step 2 CK scores to measure influence variables individually, then a regression model measured impacts of variables together.ResultsAll variables except undergraduate science GPA significantly correlated with USMLE Step 2 CK score. USMLE Step 1 had the strongest correlation (r = 0.752, p < 0.001). The regression model had an R of 0.859 with the internal medicine subject exam showing the highest beta coefficient (0.327, p < 0.001).ConclusionsThis study determined that USMLE Step 2 CK scores can be effectively predicted using available performance measures. With USMLE Step 1 becoming pass/fail in January 2022, the importance of USMLE Step 2 CK as a screening tool in the residency application process will likely increase. This study was conducted within a LInC curriculum and may have limited value in the prediction of scores within other clinical year curricula.

BackgroundMedical students must meet curricular expectations and pass national licensing examinations to become physicians. However, no previous studies explicitly modeled stages of medical students acquiring basic science knowledge. In this study, we employed an innovative statistical model to characterize students’ growth using progress testing results over time and predict licensing examination performance.MethodsAll students matriculated from 2016 to 2017 in our medical school with USMLE Step 1 test scores were included in this retrospective cohort study (N = 358). Markov chain method was employed to: 1) identify latent states of acquiring scientific knowledge based on progress tests and 2) estimate students’ transition probabilities between states. The primary outcome of this study, United States Medical Licensing Examination (USMLE) Step 1 performance, were predicted based on students’ estimated probabilities in each latent state identified by Markov chain model.ResultsFour latent states were identified based on students’ progress test results: Novice, Advanced Beginner I, Advanced Beginner II and Competent States. At the end of the first year, students predicted to remain in the Novice state had lower mean Step 1 scores compared to those in the Competent state (209, SD = 14.8 versus 255, SD = 10.8 respectively) and had more first attempt failures (11.5% versus 0%). On regression analysis, it is found that at the end of the first year, if there was 10% higher chance staying in Novice State, Step 1 scores will be predicted 2.0 points lower (95% CI: 0.85–2.81 with P < .01); while 10% higher chance in Competent State, Step 1scores will be predicted 4.3 points higher (95% CI: 2.92–5.19 with P < .01). Similar findings were also found at the end of second year medical school.ConclusionsUsing the Markov chain model to analyze longitudinal progress test performance offers a flexible and effective estimation method to identify students’ transitions across latent stages for acquiring scientific knowledge. The results can help identify students who are at-risk for licensing examination failure and may benefit from targeted academic support.

Pharmacist Licensure: Time to Step It Up?

Unintended Consequence: Diversity as a Casualty of Eliminating United States Medical Licensing Examination Step 1 Scores

The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.