Reflection and feedback in ambulatory education.

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ''problem stations'' were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.

Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high-stakes testing. This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats. We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice. Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter-panel, inter-panellist and test-retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct-irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct-irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended. Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

The undergraduate MBBS course (Bachelor of Medicine, Bachelor of Surgery) in India is designed to produce competent Indian Medical Graduates (IMGs) capable of addressing community health needs. The preparedness of these graduates for clinical practice remains a crucial area of investigation. Objective Structured Clinical Examinations (OSCEs) have emerged as a robust assessment tool for evaluating clinical competencies. This study aims to assess the feasibility and utility of the OSCE when used as an end-of-internship evaluation, describe temporal trends in OSCE performance, and analyze the correlation between OSCE scores and final-year MBBS examination results over a six-year period at a center in South India. A descriptive study was conducted at a medical college in South India, involving six batches of students (2017-2023; n = 469). OSCE stations were designed to assess cognitive, psychomotor, and affective domains. The feasibility and utility of this exercise was assessed. OSCE scores were analyzed across domains and subdomains, and correlations with final-year MBBS examination scores were explored using Pearson's correlation coefficient. Statistical significance was set at p < 0.05. The OSCE was found to be a resource-intensive evaluation, requiring 528 faculty-hours for a batch of 60 students and 843 faculty-hours for a batch of 150. The mean OSCE score was 61.3 ± 8.0%, with the highest performance in the affective domain (72.1 ± 14.2%) and the lowest in the cognitive domain (50.0 ± 15.4%). Notably, psychomotor skills were inversely correlated with cognitive skills (-0.41, p < 0.001). OSCE scores were weakly correlated with final-year MBBS examination marks (r = 0.44, p < 0.001), with stronger correlations in smaller batches (r = 0.57, p < 0.001) than larger ones (r = 0.44, p < 0.001). The COVID-19 pandemic and increased batch sizes negatively impacted clinical and technical skill performance. It is feasible to conduct OSCEs as an end of internship assessment in a medical college. OSCEs serve as a comprehensive assessment tool, underscoring the importance of multi-modal evaluation strategies. Conducting OSCEs serves as an appropriate assessment of the MBBS training programme so as to subsequently tailor the programme according to performance in various subjects, domains and subdomains.

Effective communication between doctors and patients is a central requirement for good clinical practice. The doctor's interpersonal skills contribute to patient satisfaction and compliance and can positively influence health outcomes [1]. The Royal College of Anaesthetists seeks to assess candidates' communication skills through simulated patient interviews at the primary FRCA objective structured clinical examination (OSCE). We examined the relationship between the objective OSCE mark and the subjective opinions of the participants in the interview. Candidates at the Coventry Primary FRCA Viva/OSCE course take part in mock OSCE examinations which include communication skills stations in which professional actors play the role of patients. After each such station, both the actor and the candidate were asked to complete a proforma giving their opinions of the candidates' communication skills during the interview. A total of 32 doctor:patient encounters were recorded. These were converted to percentages and compared with each other and with the OSCE mark. Comparisons showed that there was no correlation between the OSCE score and either the patient score or the candidate score suggesting that the OSCE score was a poor predictor of simulated patient satisfaction with the interpersonal and communication skills of the doctor. Mean OSCE marks were 87%, exceeding the mean scores given by simulated patients of 65% and by candidates of 77%. The OSCE scores were compared to the simulated patient scores using the paired t-test. This showed that OSCE marks significantly exceeded the simulated patient score (p < 0.0001). The mean scores given by candidates exceeded those given by actors for all criteria measuring communication skills and satisfaction. These results suggest that the OSCE station fails adequately to assess patient satisfaction with doctors' communication skills. The results also demonstrate that candidates have a falsely high opinion of how effectively they are communicating with patients. In training anaesthetists we must stress the importance of communication and there may be an argument for subjective patient input in the scoring of communication skills stations.

Introduction: In the field of medical education, the evaluation process of communication and clinical skills training is known as Objective Structured Clinical Examination (OSCE). OSCE can be assessed using the global rating method, which is a method of subjective evaluation by examiners and standardized patients of student performance. Objective: The study aims to examine the relationship between the subjectivity of examiners and standardized patients in OSCE scores using the global rating method. Methods: The study involved 985 students from the Faculty of Medicine at Maranatha Christian University who participated in the Local OSCE from December 2021 until January 2022. Secondary data were analyzed using logistic regression in SPSS software and presented in tables and multivariate analysis. Results: The global rating assessment by examiners showed a significant relationship with OSCE scores (p&lt;0.001), while the global rating assessment by standardized patients did not show a significant relationship with OSCE scores (p&gt;0.05). This indicates that the evaluation by examiners has a greater influence on OSCE results compared to the assessment by standardized patients. Recommendations: More comprehensive training is needed for standardized patients in conducting assessments using the global rating method to minimize bias. Furthermore, future research is recommended to explore other factors that may influence OSCE assessments to improve the validity and reliability of the evaluation.

BackgroundMedical students learn and practice various clinical skills during clinical clerkship. Patient encounters are important for developing clinical thinking, communication skills, and professional attitude. We investigated whether the amount of clinical experience during clerkship correlated with students’ clinical competency and students’ perception of effectiveness of their clerkship on it.MethodsFourth year medical students undertook the Objective Structured Clinical Examinations (OSCE) in August 2012. Students provided the number of patients for whom they took medical histories or performed physical examinations during clerkship and provided feedback as to whether or not the clinical clerkship was helpful in preparing OSCE. The correlation between the OSCE score and number of patients was analyzed.ResultsOne hundred thirty students completed the questionnaire (86.6%). OSCE scores correlated with the total number of patients encountered for physical examinations (correlation coefficient, 0.274; p = 0.0105). Cumulative 3-year GPAs were positively correlated with OSCE scores (correlation coefficient, 0.330; p = 0.0001). Most (92.3%) answered that their clinical clerkship was helpful in preparing them for the OSCE; however, only 20% felt that their clinical clerkship was most helpful. Others felt that role playing (38.46%) or the guide book (33.84%) was most helpful.ConclusionsThe amount of clinical experience during the students’ clerkship had a small but positive relationship with students’ clinical performance. Further research to elucidate the influence of clinical experience on clinical competency is needed.Electronic supplementary materialThe online version of this article (doi:10.1186/1472-6920-14-209) contains supplementary material, which is available to authorized users.

The script concordance test (SCT) is used in health professions education to assess a specific facet of clinical reasoning competence: the ability to interpret medical information under conditions of uncertainty. Grounded in established theoretical models of knowledge organization and clinical reasoning, the SCT has three key design features: (1) respondents are faced with ill-defined clinical situations and must choose between several realistic options; (2) the response format reflects the way information is processed in challenging problem-solving situations; and (3) scoring takes into account the variability of responses of experts to clinical situations. SCT scores are meant to reflect how closely respondents’ ability to interpret clinical data compares with that of experienced clinicians in a given knowledge domain. A substantial body of research supports the SCT's construct validity, reliability, and feasibility across a variety of health science disciplines, and across the spectrum of health professions education from pre-clinical training to continuing professional development. In practice, its performance as an assessment tool depends on careful item development and diligent panel selection. This guide, intended as a primer for the uninitiated in SCT, will cover the basic tenets, theoretical underpinnings, and construction principles governing script concordance testing.

The Objective Structured Clinical Examination (OSCE) recommended in nursing education is a new concept in practical assessment especially in psychiatric nursing. The validity of the examination is improved in OSCE as this tool tests both the process and the product with much importance to individual competency. A cross sectional study was conducted at Apollo College of Nursing, Chennai, affiliated to Tamilnadu Dr M. G. R Medical University among the, 3rd B. Sc Nursing students using simulated patients. Participants included 80 students selected through purposive sampling technique. Data was collected by evaluating the students ’performance using predetermined selfdeveloped station checklists in OSCE and answer keys in other form of examinations by the examiners. Validity was established by correlating OSCE scores with examination scores obtained by students in short answers test, and second term examination in psychiatric nursing. Collected data was entered in SPSS-16, and analyzed using appropriate descriptive and inferential statistics. OSCE scores show a high positive correlation with short answers test (0.86) and Term Examination (0.82) which was found to be statistically significant (P<0.001) indicating the good concurrent validity of OSCE. Thus this study was conducted to assess the students ’performance and validity of OSCE as a part of evaluation in Psychiatric Nursing.

Objective To explore the effect of objective structured clinical examination (OSCE) in pediatric dental education. Methods Thirty 8-year-program students and sixty 5-year-program dental students from School of Stomatology, the Fourth Military Medical University were enrolled in this study. OSCE traditional written test and the students′ satisfaction questionnaires were carried out in all students. The scores of the two testing methods applied in 2 groups were compared using SPSS 18.0 software. Results The OSCE score of the five-year-program students was 61.68 ± 5.22, while that of the eight-year-program students was 58.13 ± 6.54 (t= 0.564, P= 0.579) . And the traditional test scores of the five-year-program students and eight-year-program students were 87.90 ± 4.52 and 82.56 ± 4.44 respectively (t= 2.712, P= 0.013) . There was no significant difference in OSCE scores between the two grades. The traditional test score of the five-year-program students was higher than that of the eight-year-program students, and the scores of OSCE and the traditional test were in good agreement. 71.11% of students rated higher in OSCE. Conclusions OSCE can evaluate the comprehensive quality of students objectively, truly and comprehensively, and can achieve good results in pediatric dental education. It is worthy of further promotion. Key words: Pediatric dentistry; Education, undergraduate; Objective structured clinical examination; Eight-year-program students; Five-year-program students

ObjectivesThis study aimed to investigate the effect of clinical clerkship-associated achievements, such as performance of procedures at the student clinic, observation, and attitude towards a clerkship, on the objective structured clinical examination (OSCE) scores of dental students graduating in restorative dentistry.Materials and MethodsThe OSCEs consisted of two stations designed to assess students' clinical skills regarding cavity preparation for a class II gold inlay and a class IV composite restoration. The clerkship achievements, consisting of the number of student clinical procedures performed, observation-related OSCE, and scores of their attitudes towards a conservative dentistry clerkship, were assessed. Correlation and multiple regression analyses were conducted.ResultsThe correlation coefficient between the OSCE scores for cavity preparation for a class II gold restoration and clerkship attitude scores was 0.241 (p < 0.05). Regarding a class IV composite restoration, OSCE scores showed statistically significant correlations with the observation (r = 0.344, p < 0.01) and attitude (r = 0.303, p < 0.01) scores. In a multiple regression analysis, attitudes towards a clerkship (p = 0.033) was associated with the cavity preparation for a class II gold inlay OSCE scores, while the number of procedure observations (p = 0.002) was associated with the class IV composite restoration OSCE scores.ConclusionsThe number of clinical procedures performed by students, which is an important requirement for graduation, showed no correlation with either of the OSCEs scores.

The purpose of this study was to examine the relationship among dental students' performance on the National Board Dental Examination (NBDE Parts I and II), comprehensive written multiple-choice question examinations (MCQ examinations), and objective structured clinical examinations (OSCEs) administered at the Harvard School of Dental Medicine (HSDM). Outcomes were measured during the third and fourth years at HSDM for the classes of 2006 and 2007. Three separate OSCE exams and two comprehensive MCQ examinations are administered during years 3 and 4 at HSDM per class. The study group was made up of sixty-two students (thirty-two females and thirty males). The average NBDE Parts I and II scores of the study population were 93.32 +/-4.02 and 84.63 +/-4.25, respectively, and were associated with outcomes on all three OSCE examinations (p<or=0.044). However, using multiple regression models, the only statistically significant association occurred between NBDE Part II and OSCE 2 (p=0.003). Analysis showed that didactic predictors (NBDE Parts I and II and comprehensive MCQ examinations) explained 20.4 to 22.1 percent of the variability in OSCE scores. These results suggest that performance on OSCE examinations is not highly correlated with performance on NBDE Parts I and II and HSDM-administered MCQ examinations. The findings suggest that OSCE examinations are more likely to measure other qualities such as problem-solving ability, critical thinking, and communication skills.

Assessing clinical competence of medical students in women’s health care: use of the objective structured clinical examination

PurposeThe biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established.MethodsWe compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance.ResultsFive hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores.ConclusionWe suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.

To assess the teaching effectiveness of the Advanced Trauma Life Support (ATLS) Program among senior medical students. We used objective structured clinical examination (OSCE) and multiple choice question (MCQ) testing to assess 40 senior medical students (20 ATLS and 20 non-ATLS) at the University of Toronto. Pre- and post-ATLS, all students had four 15-minute OSCE stations and a 40-item MCQ test. The pre- and post-ATLS performance for the ATLS and non-ATLS group were compared. Cronbach's reliability coefficients were 0.81 and 0.93 for the pre- and post-ATLS OSCEs. The mean (+/- SD) OSCE scores at the four pre-ATLS OSCE stations were 7.6 +/- 2.8, 7.4 +/- 2.3, 8.3 +/- 2.7, and 10.5 +/- 3.4 for the ATLS group and 6.5 +/- 2.1, 7.0 +/- 2.2, 7.6 +/- 2.5, and 9.6 +/- 3.1 for the non-ATLS group (p = NS). Post-ATLS scores for the four OSCE stations were: 15.5 +/- 1.6, 14.1 +/- 3.2, 12.3 +/- 2.9, and 18.3 +/- 1.0 (ATLS group) and 7.9 +/- 3.5, 6.3 +/- 2.8, 7.6 +/- 2.3, and 10.9 +/- 3.3 (non-ATLS group: p < 0.01). Pre-ATLS MCQ scores were 49 +/- 9 and 52 +/- 8% for the ATLS and non-ATLS groups respectively; the ATLS group increased MCQ scores to 83 +/- 5% and the non-ATLS group did not change (53 +/- 8%). Pre-ATLS scores for adherence to priorities were: 35 +/- 14% (ATLS) and 29 +/- 13% (non-ATLS: p = NS). Post-ATLS scores were 99 +/- 6% (ATLS) and 29 +/- 19% (non-ATLS; p < 0.01). The pre-ATLS organized approach to trauma ratings (range 1 to 5) were: 1.9 +/- 1.5 (ATLS) and 1.6 +/- 0.5 (non-ATLS; p = NS) compared to post-ATLS grades of 4.6 +/- 0.4 (ATLS) and 1.7 +/- 0.5 (non-ATLS: p < 0.01). All 20 students passed the ATLS course. Using highly reliable trauma OSCE stations we have demonstrated trauma management skills acquisition by senior medical students after the ATLS course.

IntroductionThe COVID‐19 pandemic necessitated rapid adaptation of clinical competence assessments, including the transition of Objective Structured Clinical Examinations (OSCE) from in‐person to virtual formats. This study investigates the construct equivalence of a high‐stakes OSCE, originally designed for in‐person delivery, when adapted for a virtual format.MethodsA retrospective analysis was conducted using OSCE scores from the Internationally Educated Nurse Competency Assessment Program (IENCAP®). Data were collected from 15 exam administrations between January 2018 and June 2022, encompassing 2021 examinees (1936 in‐person, 85 virtual). The Many‐Facet Rasch Measurement (MFRM) model was employed to analyze the invariance of examinee ability, case difficulty, and criteria difficulty across in‐person and virtual formats.ResultsResults revealed overall examinee ability estimates remained invariant regardless of the OSCE format, while invariant violations were identified in only three of the 15 cases (N = 20%) adapted to suit the virtual format. The most significant adaptation, namely the use of a verbal physical examination to suit the virtual context achieved equivalence to its hands‐on in‐person counterpart given evidence of invariance across criteria estimates. Interestingly, criteria scores in invariant violated cases displayed a higher level of stability or consistency across the virtual OSCE formats versus their in‐person counterpart highlighting a potential benefit of the virtual versus in‐person format and potentially linked to the verbal physical examination.ConclusionThe study found that while examinee ability and case difficulty estimates exhibited some invariance between in‐person and virtual OSCE formats, criteria involving physical assessments faced challenges in maintaining construct equivalence. These findings highlight the need for careful consideration in adapting high‐stakes clinical assessments to virtual formats to ensure fairness and reliability.