Within-Session Score Gains for Repeat Examinees on a Standardized Patient Examination

Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence. Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used. The reliability of SP encounter (history taking and physical examination), PN, and communication and interpersonal skills scores were estimated with generalizability studies. Composite score reliability was estimated for varying weight combinations. Faculty were surveyed for preferred weights on the SP encounter and PN scores. Composite scores based on Kane's method were compared with weighted mean scores. Faculty suggested weighting PNs higher (60%-70%) than the SP encounter scores (30%-40%). Statistically, composite score reliability was maximized when PN scores were weighted at 40% to 50%. Composite score reliability of ICE scores increased by up to 0.20 points when SP-history taking (SP-Hx) scores were included; excluding SP-Hx only increased composite score reliability by 0.09 points. Classification accuracy for pass-fail decisions between composite and weighted mean scores was 0.77; misclassification was < 5%. Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.

So You Want to Become a Plastic Surgeon? What You Need to Do and Know to Get into a Plastic Surgery Residency

Pharmacist Licensure: Time to Step It Up?

The purpose of the present study was to assess the fit of three factor analytic (FA) models with a representative set of United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) cases and examinees based on substantive considerations. Checklist, patient note, communication and interpersonal skills, as well as spoken English proficiency data were collected from 387 examinees on a set of four USMLE Step 2 CS cases. The fit of skills-based, case-based, and hybrid models was assessed. Findings show that a skills-based model best accounted for performance on the set of four CS cases. Results of this study provide evidence to support the structural aspect of validity. The proficiency set used by examinees when performing on the Step 2 CS cases is consistent with the scoring rubric employed and the blueprint used in form assembly. These findings will be discussed in light of past research in this area.

Medical schools using formal undergrad-uate selectivity measures do so to compensate for the psychometricinadequacies of college grade-point averages, believing that moremeaning can be derived from the GPA if it is attached to a measureof institutional performance (academic rigor) or selectivity (strin-gent admission standards).Researchers have reported mixed results on whether formal mea-sures of undergraduate institution selectivity are useful contributorsto predicting medical student performance.

During the United States Medical Licensing Examination Step 2 Clinical Skills exam, examinees rotate through 12 standardized patient (SP) encounters. SPs rate examinees on Communication and Interpersonal Skills (CIS). Examinees have at most 15 minutes to interact with each SP, but can end the encounter sooner. The current work assesses the relationship between CIS ratings and time examinees spend interacting with the SP. A total of 5,955 encounters from a fall 2007 sample of examinees were time-stamped to indicate total encounter time and closure time. Hierarchical linear modeling was employed to assess how well total and closure time, in conjunction with other examinee- and SP-case-related variables, predicted CIS scores. Time spent on closure had a larger impact on CIS scores than overall time used. Other variables, such as examinee gender, Spoken English Proficiency, and SP stringency, were also significantly related to CIS scores. Both time measures were significantly positively related to the CIS outcome variable, indicating a link between time used and SP satisfaction with communication, with closure time having the larger coefficient. This finding suggests that a good closure may be a lengthy one.

Determine whether the National Board of Medical Examiners (NBME) Subject Examination performance from six clerkships correlated with United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) Examination scores. Also, examine correlations between medical students' preclinical and clinical year mean cumulative grade point average (GPA), subject exam, and USMLE performance. The sample consisted of 507 students from the 2008-2010 graduating classes from the authors' medical school. Pearson correlations followed by stepwise linear regressions were used to investigate variance in USMLE Steps 1 and 2 CK scores explained by subject exam scores and GPA. Data from 484 (95.5%) students were included. USMLE Steps 1 and 2 CK scores had moderate-to-large positive correlations with all subject exam scores and with both GPA variables. Correlations between composite subject exam scores and USMLE Steps 1 and 2 CK exams were 0.69 and 0.77, respectively. Regression analysis demonstrated that subject exams and GPA accounted for substantial variance in Steps 1 and 2 CK exam scores (62% and 61%); when entered into the regression model first, primary care clerkship subject examination scores accounted for most of this variance. The moderate-to-large correlations between subject exam performance and USMLE scores provide reassurance that subject exam scores are associated with USMLE performance. Furthermore, the considerable variance in USMLE scores accounted for by primary care NBME scores may be due to primary care topics being reinforced through all clerkships and comprising a significant portion of the USMLE examinations, particularly Step 2 CK.

The unexpected discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam in January 2021 carries both risks and opportunities for medical education in the United States. Step 2 CS had far-reaching effects on medical school curricula and school-based clinical skills assessments. Absent the need to prepare students for this high-stakes exam, will the rigor of foundational clinical skills instruction and assessment remain a priority at medical schools? In this article, the authors consider the potential losses and gains from the elimination of Step 2 CS and explore opportunities to expand local summative assessments beyond the narrow bounds of Step 2 CS. The responsibility for implementing a rigorous and credible summative assessment of clinical skills that are critical for patient safety as medical students transition to residency now lies squarely with medical schools. Robust human simulation (standardized patient) programs, including regional and virtual simulation consortia, can provide infrastructure and expertise for innovative and creative local assessments to meet this need. Novel applications of human simulation and traditional formative assessment methods, such as workplace-based assessments and virtual patients, can contribute to defensible summative decisions about medical students' clinical skills. The need to establish validity evidence for decisions based on these novel assessment methods comprises a timely and relevant focus for medical education research.

Training Standardized Patients for a High-Stakes Clinical Performance Examination in the California Consortium for the Assessment of Clinical Competence

Lecture versus standardized patient interaction in the surgical clerkship: a randomized prospective cross-over study

Communication skills training is a routine practice in medical education designed to instruct and evaluate future physicians in matters of patient-provider interaction. Based on the United States Medical Licensing Examination Step 2 Clinical Skills (CS), medical schools across the United States hire and train standardized patients (SPs) to act as patients in and evaluators of simulated interactions with medical students (MSs). Using discourse analysis, I examine how a computerized assessment form creates a particularized version of communication skills with implications for future practice. The 39-item checklist is completed by SPs following a simulated interaction designed to prepare third-year MSs for the Step 2 CS. Specifically, I analyze how the form is structured to make recognizable specific communication skills tasks, who should complete said tasks, and what varying degrees of communication skills competency are within the realm of task completion. By analyzing the form, I consider the agency of texts in medical education, the implications of technologizing communication as an institutional skill, and the limitations of enlisting SPs to evaluate communication skills competency under the guise of a patient perspective.

Unintended Consequence: Diversity as a Casualty of Eliminating United States Medical Licensing Examination Step 1 Scores

The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.

The purpose of this study was to compare residents in the integrated and independent tracks of plastic surgery residency training using several quantitative educational metrics. The Harvard Combined Plastic Surgery Residency Training Program offers an ideal model for this comparison, as it is the largest in the country and operates both tracks simultaneously. The authors compared several quantitative educational metrics of all 22 independent and 24 integrated residents matched into the Harvard Program since its inception. These quantitative educational metrics include medical school reputation, United States Medical Licensing Examination (USMILE, Philadelphia, Pa.) step 1 scores, publication productivities during different educational periods (preresidency and pre-plastic surgery training), in-service scores, exit rankings (overall and operative skills), and percentage of M.D./Ph.D. residents. Integrated residents graduated from more highly ranked medical schools than independent residents (mean tier, 1.5 versus 2.3; p = 0.0345), had higher United States Medical Licensing Examination step 1 scores (mean, 235 versus 220; p = 0.0234), and had higher preresidency publication scores (mean, 68 versus 14; p = 0.015). There are more M.D./Ph.D. residents in the integrated than in the independent track (33.3 percent versus 4.4 percent; p = 0.027). Independent residents had higher pre-plastic surgery training publication scores (mean, 77 versus 6; p = 0.0003). In-training quantitative educational metrics were similar between the two tracks (p > 0.05 in all cases). To address potential caveats of the study, all publication scores were compared with and without correction for the presence of M.D./Ph.D. residents, which yielded similar results. The authors also compared the match ranks of all their residents and found them to be statistically comparable between the two tracks. In the Harvard Program, the integrated residents had better preresidency quantitative educational metrics, whereas the independent residents had better pre-plastic surgery training quantitative educational metrics. In training quantitative educational metrics were remarkably similar between the two tracks. In conclusion, this study, a systematic comparison between the two tracks, offers not only a valid foundation for future study of long-term outcome results but also a starting point from which to improve on the educational experience of residents in either track.

BackgroundUnited States Medical Licensing Examination (USMLE) Step 1 is a common metric looked at by residency programs to determine invitations for candidates to interview. However, USMLE Step 2 Clinical Knowledge (CK) has also been an important factor for selecting applicants to interview and plays a significant role during applicant selection. This study aims to identify academic performance measures that correlate with USMLE Step 2 CK scores and to develop a model to predict USMLE Step 2 CK scores using previous academic measures from the first two cohorts in the longitudinal interleaved clerkship (LInC) at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM).SettingThe KSOM is a newly accredited US allopathic medical school that accepted its first class in 2017. At KSOM, a LInC model is used in the primary clinical year. In this model, rotations are two weeks in duration before moving on to the next specialty. Students complete the National Board of Medical Examiners (NBME) subject examinations in all six specialties in one week at the midpoint and the end of the LInC. Students who passed an exam at the midpoint can opt out of that exam at the end as the higher of the two exam scores is recorded. However, most students choose to take all the exams again to improve their scores and prepare for USMLE Step 2 CK.MethodologyAcademic performance measures were gathered from the class of 2021 and 2022 (n = 101) including undergraduate grade point average (GPA), undergraduate science GPA, medical college admission test score, USMLE Step 1 score, NBME clinical subject exam scores, and USMLE Step 2 CK scores. Pearson correlations were run between the performance variables and USMLE Step 2 CK scores to measure influence variables individually, then a regression model measured impacts of variables together.ResultsAll variables except undergraduate science GPA significantly correlated with USMLE Step 2 CK score. USMLE Step 1 had the strongest correlation (r = 0.752, p < 0.001). The regression model had an R of 0.859 with the internal medicine subject exam showing the highest beta coefficient (0.327, p < 0.001).ConclusionsThis study determined that USMLE Step 2 CK scores can be effectively predicted using available performance measures. With USMLE Step 1 becoming pass/fail in January 2022, the importance of USMLE Step 2 CK as a screening tool in the residency application process will likely increase. This study was conducted within a LInC curriculum and may have limited value in the prediction of scores within other clinical year curricula.