Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review.

Background and Aims:National Medical commission of India (NMC) has introduced Competency based Medical Education (CBME) following the international trend. Competency based assessment differs from traditional assessment and we need to adopt to newer work place-based assessments (WPBA). Direct Observation of Procedural Skills (DOPS) is one such assessment tool, which assesses procedural skills of a student. Hence this study was conducted to evaluate the effectiveness of DOPS among Anesthesiology postgraduate students.Material and Methods:A prospective interventional study was conducted, where in 55 postgraduate students underwent DOPS assessment while performing three routine core skills of Anesthesiology. Observations were documented (Pre-DOPS-Score) and immediate verbal feedback was given. Students were educated through demonstration of all three procedures on actual patients by three faculty members using the DOPS checklist. The students were finally assessed again with proforma and DOPS scores were documented (Post-DOPS Score). Mean Pre-DOPS and Post-DOPS assessment Scores were compared. Student and faculty Perceptions were taken regarding the DOPS assessment.Results:There was significant improvement in the procedural skill performance after DOPS with a statistically significant difference. Average time taken for DOPS test was 11 min. Average time taken for giving feedback was 13 min. Students and faculty members gave positive feedback about DOPS.Conclusion:DOPS is very feasible and effective assessment tool, improves procedural skills of Anaesthesiology postgraduate students, helps to develop good clinical skills which finally brings good clinical care.

Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools. To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty. This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada. Introduction of the Competency-Based Achievement System (CBAS). Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program. Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183). The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.

Workplace-based assessment (WPBA) is an increasingly important part of postgraduate medical training and its results may be used as evidence of professional competence. This study evaluates the ability of WPBA to distinguish UK Foundation Programme (FP) doctors with training difficulties and its effectiveness as a surrogate marker for deficiencies in professional competence. We conducted a retrospective observational study using anonymised records for 1646 trainees in a single UK postgraduate deanery. Data for WPBAs conducted from August 2005 to April 2009 were extracted from the e-portfolio database. These data included all scores submitted by trainees in FP years 1 and 2 on mini-clinical evaluation exercise (mini-CEX), case-based discussion (CbD), direct observation of procedural skills (DOPS) and mini-peer assessment tool (mini-PAT) assessments. Records of trainees in difficulty, as identified by their educational supervisors, were tagged as index cases. Main outcome measures were odds ratios (ORs) for associations between mean WPBA scores and training difficulties. Further analyses by the reported aetiology of the training difficulty (health-, conduct- or performance-related) were performed. Of the 1646 trainees, 92 had been identified as being in difficulty. Mean CbD and mini-CEX scores were lower for trainees in difficulty and an association was found between identified training difficulties and average scores on the mini-CEX (OR = 0.54; p = 0.034) and CbD (OR = 0.39; p = 0.002). A receiver operator characteristic curve analysis of mean WPBA scores for diagnosing 'in difficulty' status yielded an area under the curve of 0.64, indicating weak predictive value. There was no statistical evidence that mean scores on DOPS and mini-PAT assessments differed between the two groups. Analysis of a large dataset of WPBA scores revealed significant associations between training difficulties and lower mean scores on both the mini-CEX and CbD. Models show that using WPBA scores is, however, not a valid way of screening for trainees in difficulty. Workplace-based assessments have value as formative assessments that prompt supervision, feedback and reflection. They should not be relied upon to certify competence and their use for such ends may reduce their effectiveness in training. Their results should be interpreted in the context of multiple other methods of assessment, with the aim of achieving a genuinely holistic and representative assessment of professional competence.

There is a lack of studies that have systematically synthesized the existing literature related to the competency-based medical education (CBME) assessment process and identified gaps in existing knowledge about the assessment process structure. The aims of this study are as follows: 1. A systematic review of the assessment framework of CBME 2. Designing a model framework of assessment for CBME that will be relevant in the Indian context. The databases searched will be PubMed, MEDLINE (Ovid), EMBASE (Ovid), Scopus, Web of Science, and Google Scholar. General limits to the search will be set in the English language, published between January 2006 and December 2020, specific to competency-based education and assessment methods. After initial screening for duplicates and search limits, the title and abstract of the remaining studies will be screened. Subsequently, the full text of the articles will be screened against the inclusion and exclusion criteria for final selection. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of the systematic review will be used to describe the selection of the studies. The article quality will be assessed using the quality assessment tool for diverse study designs (QATSDD). The data synthesis will start with a descriptive summary (in tabular form) of the included studies. This study will try to provide comprehensive data to address challenges related to the CBME assessment process, which can help in evolving the robust assessment framework of CBME.

Assessment and feedback methods in competency-based medical education

Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions. We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments. We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively. Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions. The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders. In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain. Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Competency-based medical education (CBME) is both an educational philosophy and an approach to educational design. CBME has already had a broad impact on medical schools, residency programs, and continuing professional development in health professions around the world. As the CBME movement evolves and CBME programs are implemented, a wide range of emerging research questions will warrant scholarly examination. In this paper, we describe a proposed CBME research agenda developed by the International CBME Collaborators. The resulting framework includes questions about the meaning of key concepts of CBME and their implications for learners, faculty members, and institutional structures. Other research questions relate to the learning process, the meaning of entrustment decisions, fundamental measurement issues, and the nature and definition of standards. The exploration of these questions will help to solidify the theoretical foundation of CBME, but many issues related to implementation also need to be addressed. These pertain to, among other things, nurturing independent learning, assembling and using assessment results to make decisions about competence, structuring feedback, supporting remediation, and how best to evaluate the longer-term outcomes of CBME. High-quality research on these questions will require rigorous outcome measures with strong validity evidence. The complexity of CBME necessitates theoretical and methodological diversity. It also requires multi-institutional studies that examine effects at multiple levels, from the learner to the team, the institution, and the health care system. Such a framework of research questions can guide and facilitate scholarly discourse on the theoretical and practical body of knowledge related to competency-based health professions education.

The healthcare workforce plays a pivotal role in cancer care delivery, leadership, policy, education, and research in complex cancer systems. To ensure quality and relevance, health professionals must have the necessary competencies to deliver patient-centered and efficient care, coupled with the ability to work in teams and manage health resources wisely. This paper aims to review the concept of competency-based medical education (CBME) in the context of oncology to provide insights and guidance for those interested in adopting or adapting competency-based education in training programs. The results of a scoping review of CBME in oncology are presented here to describe the current status of CBME in oncology. The literature describing the implementation and evaluation of CBME in oncology training programs for medical professionals internationally is summarized and key themes identified to provide practical guidance for educators. Further, the paper identifies critical competencies for oncology education and training globally and presents recommendations and opportunities for collaboration in competency-based education and training in oncology. The authors argue for increased global collaboration and networking in the realm of CBME to facilitate the establishment of a competent global cancer care workforce.

Many of those involved in continuing professional development (CPD) over the past 10 years have engaged in discussions about its goals and activities. Whereas in the past CPD was viewed as an education intervention directed towards the medical expert role, recent research highlights the need to expand the scope of CPD and to promote its more explicit role in improving patient care and health outcomes. Recent developments in quality improvement (QI) and competency-based medical education (CBME), guided by appropriate theories of learning and change, can shed light on how the field might best advance. This paper describes principles of QI and CBME and how they might contribute to CPD, explores theoretical perspectives that inform such an integration and suggests a future model of CPD. Continuing professional development seeks to improve patient outcomes by increasing physician knowledge and skills and changing behaviours, whereas QI takes the approach of system and process change. Combining the strengths of a CPD approach with strategies known to be effective from the field of QI has the potential to harmonise the contributions of each, and thereby to lead to better patient outcomes. Similarly, competency-based CPD is envisioned to place health needs and patient outcomes at the centre of a CPD system that will be guided by a set of competencies to enhance the quality of practice and the safety of the health system. We propose that the future CPD system should adhere to the following principles: it should be grounded in the everyday workplace, integrated into the health care system, oriented to patient outcomes, guided by multiple sources of performance and outcome data, and team-based; it should employ the principles and strategies of QI, and should be taken on as a collective responsibility by physicians, CPD provider organisations, regulators and the health system.

Evaluating the effect of assessment form design on the quality of feedback in one Canadian ophthalmology residency program as an early adopter of CBME

Although competency-based medical education has become the standard for physician training in the West, many developing countries have not yet adopted competency-based training. In 2009 in the United Arab Emirates, the government regulatory and operational authorities for healthcare in Abu Dhabi mandated a wide-scale reform of the emirate’s postgraduate residency programs to the competency-based framework of the newly formed Accreditation Council for Graduate Medical Education-International (ACGME-I). This article briefly describes the rationale for competency-based medical education and provides an overview of the transition from traditional, time-based residency training to competency-based postgraduate medical education for the Pediatrics residency programs in Abu Dhabi. We will provide data on the initial impact of this transition on resident performance and patient outcomes in a Pediatrics residency program in an academic medical center in the United Arab Emirates.

IntroductionData supporting milestone development during flexible sigmoidoscopy (FS) training are lacking. We aimed to present validity evidence for our formative direct observation of procedural skills (DOPS) assessment in FS, and...

Objective: The American Association of Colleges of Nursing has identified competency-based education (CBE) as a priority in nursing education. The Objective Structured Clinical Examination (OSCE) has been used across health professions as a tool to incorporate competency-based education. However, the OSCE has been correlated with intensive faculty resources and high costs. The objective of this study was to discuss the evaluation of a low cost OSCE and its ability to incorporate the role of self-assessment in competency-based education within a nurse practitioner program.Methods: Faculty at a public university developed and evaluated an OSCE, exploring its implementation as a component of CBE while minimizing costs using a quasi-experimental design. Nine nurse practitioner students in their third year of a BSN-DNP program completed a pre- and post- assessment of their perceived ability in three OSCEs. Undergraduate nursing students were recruited for the standardized patient role. The OSCEs were recorded for evaluation by faculty and for self-evaluation by the students.Results: There was no significant difference noted in student self-evaluations pre- and post-assessment. There was a statistical difference in the faculty ratings of the student in the otitis media OSCE, with the student rating their performance higher than faculty. There was no statistical difference noted in either the women’s health or hypertension assessments.Conclusions: Recordings of the OSCEs allowed students to identify strengths and weaknesses, cultivating the practice of self-assessment. Integration of minimal cost OSCEs provides opportunities for programs with varying budgets to incorporate it as a component of CBE.

Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation. Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community. The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised. The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.