Medical Council of Canada

Purpose In the post-COVID era, recognizing evolving physician competencies is crucial for guiding medical education and test development. This study aimed to extract valuable insights concerning emerging physician competencies from influencers’ posts on X, leveraging an AI-driven approach. Method Two datasets pertaining to medical competency were analyzed, with posts collected from January 1, 2020, to June 1, 2023. Social network analyses were performed to identify influencers leading medical competency conversations on X. ChatGPT was utilized for textual analyses of influencers’ posts to reveal core themes of physician competencies. Results Social network analysis revealed that medical professionals played a predominant role in disseminating information on medical competency on X. Textual analysis identified six core themes in the CanMEDS dataset—clinical learning environment, anti-racism, EDI, adaptive expertise, planetary health, and leadership development—and seven in the MedEd dataset—cultural competency, structural competency, assessment models, virtual care, EDI, leadership development, and wellness. Conclusion The identified themes emphasize physicians’ competencies in addressing health disparities, preparing for real-world challenges, adapting to the evolving healthcare landscape, and leading effectively in diverse healthcare settings. The findings hold significant implications for medical education, test development, and the integration of artificial intelligence in physician competency assessment.

Abstract Background The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. Reliability and validity evidence for the modified NAC Examination is presented. This paper outlines validity and reliability evidence for modifications required due to significant interruptions or changes to OSCEs. Methods Cronbach’s alpha, decision consistency, and accuracy values were calculated to quantify measurement error. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs non-CSA), postgraduate training (PGT) (no PGT vs PGT), and language of examination (English vs French). Criterion relationships were summarized using correlations within and between NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions The NAC Examination yields reliable total scores and pass/fail decisions, expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that had major modifications and interruptions.

Consensus methods such as the Delphi and nominal group techniques are increasingly utilized within medical education research. This educator's blueprint paper provides practical strategies regarding five key steps for ensuring best practices when using consensus methods. These strategies include deciding which consensus method is best, developing the initial questionnaire, identifying the participants, determining the number of rounds and consensus threshold, and describing and justifying any modifications. These strategies will help guide education researchers on their next study using consensus methods.

Newer electronic differential diagnosis supports (EDSs) are efficient and effective at improving diagnostic skill. Although these supports are encouraged in practice, they are prohibited in medical licensing examinations. The purpose of this study is to determine how using an EDS impacts examinees' results when answering clinical diagnosis questions. The authors recruited 100 medical students from McMaster University (Hamilton, Ontario) to answer 40 clinical diagnosis questions in a simulated examination in 2021. Of these, 50 were first-year students and 50 were final-year students. Participants from each year of study were randomised into one of two groups. During the survey, half of the students had access to Isabel (an EDS) and half did not. Differences were explored using analysis of variance (ANOVA), and reliability estimates were compared for each group. Test scores were higher for final-year versus first-year students (53 ± 13% versus 29 ± 10, p < 0.001) and higher with the use of EDS (44 ± 28% versus 36 ± 26%, p < 0.001). Students using the EDS took longer to complete the test (p < 0.001). Internal consistency reliability (Cronbach's alpha) increased with EDS use among final-year students but was reduced among first-year students, although the effect was not significant. A similar pattern was noted in item discrimination, which was significant. EDS use during diagnostic licensing style questions was associated with modest improvements in performance, increased discrimination in senior students and increased testing time. Given that clinicians have access to EDS in routine clinical practice, allowing EDS use for diagnostic questions would maintain ecological validity of testing while preserving important psychometric test characteristics.

Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education. The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process. A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge). Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use.

IntroductionThe COVID‐19 pandemic has caused unprecedented stress to the medical education community, potentially worsening problems like burnout and work‐life imbalance that its members have long been grappling with. However, the collective struggle sparked by the pandemic could generate the critical reflection necessary for transforming professional values and practices for the better. In this hermeneutic phenomenological study, we explore how the community is adapting—and even reconceptualising—their personal and professional roles amidst the COVID‐19 crisis.MethodBetween April and October 2020, we conducted 27 (17F, 10M) semi‐structured interviews with medical trainees (8), physicians (8), graduate students (3) and PhD scientists (8) working in medical education in Canada, the United States and Switzerland. Data analysis involved a variety of strategies, including coding for van Manen's four lifeworld existentials, reflexive writing and multiple team meetings.ResultsParticipants experienced grief related to the loss of long‐established personal and professional structures and boundaries, relationships and plans for the future. However, experiences of grief were often conflicting. Some participants also experienced moments of relief, perceiving some losses as metaphorical permissions slips to slow down and focus on their well‐being. In turn, many reflected on the opportunity they were being offered to re‐imagine the nature of their work.DiscussionParticipants' experiences with grief, relief and opportunity resonate with Ratcliffe's account of grief as a process of relearning the world after a significant loss. The dismantling of prior life structures and possibilities incited in participants critical reflection on the nature of the medical education community's professional practices. Participants demonstrated their desire for more flexibility and autonomy in the workplace and a re‐adjustment of the values and expectations inherent to their profession. On both individual and systems levels, the community must ensure that long‐standing calls for wellness and work‐life integration are realised—and persist—after the pandemic is over.

Objectives: Since 2005, the International Committee of Medical Journal Editors (ICMJE) requires randomized controlled trials registration on a public platform as a condition for publication. Study outcomes must be disclosed before enrolment of the first patient. Any protocol modification should be duly declared and justified thereafter. We aimed to evaluate if outcomes of phase III trials regarding the novel widely-prescribed antithrombotic agents were duly registered. To do so, we searched for major registration issues by assessing the data found on the public platform ClinicalTrials.gov and in the corresponding publications.

Error is inevitable in medicine, given its inherent uncertainty and complexity. Errors can teach powerful lessons; however, because of physicians' self-imposed silence and the intricacies of responsibility and blame, learning from medical error has been underexplored. The purpose of this study was to understand how physicians perceived learning from medical errors by exploring the tension between responsibility and blame and factors that affected physicians' learning. Nineteen physicians participated in semistructured interviews, conducted in 2016-2017 at Western University in Canada, that probed their experiences in learning from medical errors. Data collection and analysis were conducted iteratively, with themes identified through constant comparative analysis. Participants felt personal responsibility and blame for their errors. Residency produced particularly salient memories of errors. Participants identified interconnecting cultural factors (normalizing error, peer support and mentorship, formal rounds) and individual factors (emotional response, confidence and experience), which either helped or hindered their perceived learning. Learning from medical error requires navigation through blame and responsibility. The keen responsibility felt by physicians must be acknowledged when enacting a system-based approach to medical error. Adopting a learning culture perspective suggests opportunities to enable and disable features of the learning environment to optimize learning from error as residents learn to become the most responsible physician for all outcomes. A better understanding of the factors that shape learning from error can help make the transition from error to learning more explicit, thereby increasing the opportunity to learn and teach from errors that permeate the practice of medicine.