Systematic Reviews in Medical Education.

Purpose: Descriptive psychopathology (DP, sometimes called psychopathology or phenomenology) is the language of psychiatry and is dedicated to the description of mental symptoms. Due to its importance, there is an ongoing case to put it back at the heart of psychiatry and its training. This study seeks to examine the literature on how to train psychiatry residents in DP, including reported educational interventions and educational methods. Method: The authors conducted a systematic review following the PRISMA and BEME guidelines to identify literature on how to train psychiatry residents in DP. In May 2019, they searched in Embase, ERIC, PsycINFO, PubMed, Scopus, and Web of Science; of 7,199 initial results, 26 sources were finally included for analysis. The assessment tools were the CRAAP test, Kirkpatrick’s 4 levels, and (when applicable) the Medical Education Research Study Quality Instrument (MERSQI). Results: The mean CRAAP score was 38.885 of a possible 50 (SD 0.983; range: 36.859–40.910). Fourteen sources (53.8%) had some kind of training evaluation: Kirkpatrick’s level 1 was present in nearly all (13) and was the highest in half of them (7). Regarding the educational interventions, the mean MERSQI score was 10.592 of a possible 18 (SD 2.371; range 9.085–12.098). Lectures were the most widely reported educational method (5); among those in clinical settings, the live supervised interview with feedback was the most usual (4). Conclusions: Despite its core importance as the language of psychiatry, the literature about training psychiatry residents in DP is scarce and heterogeneous. General lack of training evaluation and ongoing overemphasis on Kirkpatrick’s levels 1–2 at the expense of levels 3–4 are causes for concern. During the review process, the authors identified a selection of educational interventions that could serve as the basis for the design of new training efforts in both clinical and nonclinical settings. Topics for future research are also suggested, such as the role of DP in competency-based training frameworks now in vogue and a series of neglected contents. Finally, the combined use of the CRAAP test and the MERSQI may be useful for future systematic reviews in medical education.

Background: The medical education research study quality instrument (MERSQI) was designed to appraise medical education research quality based on study design criteria. As with many such tools, application of the results may have unintended consequences. This study applied the MERSQI to published medical education research identified in a bibliometric analysis. Methods: A bibliometric analysis identified highly cited articles in medical education that two authors independently evaluated using the MERSQI. After screening duplicate or non-research articles, the authors reviewed 21 articles with the quality instrument. Initially, five articles were reviewed independently and results were compared to ensure agreed upon understanding of the instrument items. The remainder of the articles were independently reviewed. Overall scores for the articles were analyzed with a paired samples t-test and individual item ratings were analyzed for inter-rater reliability. Results: There was a significant difference in mean MERSQI score between reviewers. Inter-rater reliability for MERSQI items labeled response rate, validity and outcomes were considered unacceptable. Conclusions: Based on these results there is evidence that MERSQI items can be significantly influenced by interpretation, which lead to a difference in scoring. The MERSQI is a useful guide for identifying research methodologies. However, it should not be used to make judgments on the overall quality of medical education research methodology in its current format. The authors make specific recommendations for how the instrument could be revised for greater clarity and accuracy.

Despite the importance of pain management across specialties and the effect of poor management on patients, many physicians are uncomfortable managing pain. This may be related, in part, to deficits in graduate medical education (GME). We sought to evaluate the methodological rigor of and summarize findings from literature on GME interventions targeting acute and chronic non-cancer pain management. We conducted a systematic review by searching PubMed, MedEdPORTAL, and ERIC (Education Resources Information Center) to identify studies published before March 2019 that had a focus on non-cancer pain management, majority of GME learners, defined educational intervention, and reported outcome. Quality of design was assessed with the Medical Education Research Study Quality Instrument (MERSQI) and Newcastle-Ottawa Scale-Education (NOS-E). One author summarized educational foci and methods. The original search yielded 6149 studies; 26 met inclusion criteria. Mean MERSQI score was 11.6 (SD 2.29) of a maximum 18; mean NOS-E score was 2.60 (SD 1.22) out of 6. Most studies employed a single group, pretest-posttest design (n=16, 64%). Outcomes varied: 6 (24%) evaluated reactions (Kirkpatrick level 1), 12 (48%) evaluated learner knowledge (level 2), 5 (20%) evaluated behavior (level 3), and 2 (8%) evaluated patient outcomes (level 4). Interventions commonly focused on chronic pain (n=18, 69%) and employed traditional lectures (n=16, 62%) and case-based learning (n=14, 54%). Pain management education research in GME largely evaluated chronic pain management interventions by assessing learner reactions or knowledge at single sites.

IntroductionThis systematic review aimed to describe the assessment practices related to self‐directed learning (SDL) in undergraduate health professions education. There were three main objectives of the review: 1) evaluating the different types of assessment used to determine the effectiveness of SDL; 2) assessing the effectiveness of SDL methods on student learning; and 3) exploring how readiness for SDL is assessed.MethodsThe PRISMA standards were followed for this systematic review. A total of seven electronic databases were searched: CINAHL, Embase, ERIC, PsycINFO, PubMed, Scopus, and Web of Science. Articles published in English between 2015 and March 2020 were retrieved for further evaluation. Two independent investigators screened abstracts and full‐text articles using the inclusion and exclusion criteria. The same investigators independently extracted data from included articles: the type of participant, sample size, study design, SDL activities, and assessment methods used to evaluate effectiveness of SDL. In addition, two independent investigators evaluated study quality using the validated Medical Education Research Study Quality Instrument (MERSQI). A third investigator resolved discrepancies and disagreements.ResultsA total of 65 studies from nineteen different countries were included in the analysis. Most study participants were medical (N=33) or nursing (N=20) students. Overall, there were wide variations in the definition of SDL, SDL activities used, and in the assessments developed to evaluate the effectiveness of SDL activities. The most common type of assessment used was a student perception survey, unrelated to a SDL readiness assessment. Internally‐developed knowledge exams were also commonly used. A total of 38 studies concluded that the SDL had a positive outcome, measured by the assessment instrument, whereas 21 studies reported a mixed or neutral outcome and six studies reported a negative outcome.ConclusionsThis systematic review demonstrates the wide variations in both the SDL activities used as well as the assessment methods developed to determine SDL effectiveness. Since SDL is a necessary skill for lifelong learning and success as a healthcare professional, it must be used and assessed appropriately. It may be valuable to standardize SDL activities and assessment methods across healthcare professions education. In addition, future research should be conducted to help establish the most accurate and effective assessment methods for SDL.

High-quality research into education costs can inform better decision making. Improvements to cost research can be guided by information about the research questions, methods and reporting of studies evaluating costs in health professions education (HPE). Our objective was to appraise the overall state of the field and evaluate temporal trends in the methods and reporting quality of cost evaluations in HPE research. We searched the MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), EMBASE, Business Source Complete and ERIC (Education Resources Information Centre) databases on 31 July 2017. To evaluate trends over time, we sampled research reports at 5-year intervals (2001, 2006, 2011 and 2016). All original research studies in HPE that reported a cost outcome were included. The Medical Education Research Study Quality Instrument (MERSQI) and the BMJ economic checklist were used to appraise methodological and reporting quality, respectively. Trends in quality over time were analysed. A total of 78 studies were included, of which 16 were published in 2001, 15 in 2006, 20 in 2011 and 27 in 2016. The region most commonly represented was the USA (n=43). The profession most commonly referred to was that of the physician (n=46). The mean±standard deviation (SD) MERSQI score was 10.9±2.6 out of 18, with no significant change over time (p=0.55). The mean±SD BMJ score was 13.5±7.1 out of 35, with no significant change over time (p=0.39). A total of 49 (63%) studies stated a cost-related research question, 23 (29%) stated the type of cost evaluation used, and 31 (40%) described the method of estimating resource quantities and unit costs. A total of 16 studies compared two or more interventions and reported both cost and learning outcomes. The absolute number of cost evaluations in HPE is increasing. However, there are shortcomings in the quality of methodology and reporting, and these are not improving over time.

BackgroundThis paper is an up-to-date systematic review on educational interventions addressing history taking. The authors noted that despite the plethora of specialized training programs designed to enhance students‘ interviewing skills there had not been a review of the literature to assess the quality of each published method of teaching history taking in undergraduate medical education based on the evidence of the program’s efficacy.MethodsThe databases PubMed, PsycINFO, Google Scholar, opengrey, opendoar and SSRN were searched using key words related to medical education and history taking. Articles that described an educational intervention to improve medical students’ history-taking skills were selected and reviewed. Included studies had to evaluate learning progress. Study quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI).ResultsSeventy-eight full-text articles were identified and reviewed; of these, 23 studies met the final inclusion criteria. Three studies applied an instructional approach using scripts, lectures, demonstrations and an online course. Seventeen studies applied a more experiential approach by implementing small group workshops including role-play, interviews with patients and feedback. Three studies applied a creative approach. Two of these studies made use of improvisational theatre and one introduced a simulation using Lego® building blocks. Twenty-two studies reported an improvement in students’ history taking skills. Mean MERSQI score was 10.4 (range 6.5 to 14; SD = 2.65).ConclusionsThese findings suggest that several different educational interventions are effective in teaching history taking skills to medical students. Small group workshops including role-play and interviews with real patients, followed by feedback and discussion, are widespread and best investigated. Feedback using videotape review was also reported as particularly instructive. Students in the early preclinical state might profit from approaches helping them to focus on interview skills and not being distracted by thinking about differential diagnoses or clinical management. The heterogeneity of outcome data and the varied ways of assessment strongly suggest the need for further research as many studies did not meet basic methodological criteria. Randomized controlled trials using external assessment methods, standardized measurement tools and reporting long-term data are recommended to evaluate the efficacy of courses on history taking.

BackgroundDeficiencies in medical education research quality are widely acknowledged. Content, internal structure, and criterion validity evidence support the use of the Medical Education Research Study Quality Instrument (MERSQI) to measure education research quality, but predictive validity evidence has not been explored.ObjectiveTo describe the quality of manuscripts submitted to the 2008 Journal of General Internal Medicine (JGIM) medical education issue and determine whether MERSQI scores predict editorial decisions.Design and ParticipantsCross-sectional study of original, quantitative research studies submitted for publication.MeasurementsStudy quality measured by MERSQI scores (possible range 5–18).ResultsOf 131 submitted manuscripts, 100 met inclusion criteria. The mean (SD) total MERSQI score was 9.6 (2.6), range 5–15.5. Most studies used single-group cross-sectional (54%) or pre-post designs (32%), were conducted at one institution (78%), and reported satisfaction or opinion outcomes (56%). Few (36%) reported validity evidence for evaluation instruments. A one-point increase in MERSQI score was associated with editorial decisions to send manuscripts for peer review versus reject without review (OR 1.31, 95%CI 1.07–1.61, p = 0.009) and to invite revisions after review versus reject after review (OR 1.29, 95%CI 1.05–1.58, p = 0.02). MERSQI scores predicted final acceptance versus rejection (OR 1.32; 95% CI 1.10–1.58, p = 0.003). The mean total MERSQI score of accepted manuscripts was significantly higher than rejected manuscripts (10.7 [2.5] versus 9.0 [2.4], p = 0.003).ConclusionsMERSQI scores predicted editorial decisions and identified areas of methodological strengths and weaknesses in submitted manuscripts. Researchers, reviewers, and editors might use this instrument as a measure of methodological quality.Electronic supplementary materialThe online version of this article (doi:10.1007/s11606-008-0664-3) contains supplementary material, which is available to authorized users.

Flipped classroom (FC) instruction has become increasingly common in graduate medical education (GME). The purpose of this study was to profile the use of FC in the GME setting and assess the current status of research quality. We conducted a systematic literature search of major health and social science databases from July 2017 to July 2018. Articles were screened to ensure they described use of the FC method in an Accreditation Council for Graduate Medical Education-accredited residency program and included research outcomes. Resulting articles were analyzed, described, and evaluated for research quality using the Kirkpatrick framework and the Medical Education Research Study Quality Instrument (MERSQI). Twenty-two articles were identified, all of which were recently published. Five were only indirectly related to FC methods. Most studies reported Kirkpatrick-level outcomes. Studies involving resident learner opinions were generally positive. Pre-posttest studies resulted in large positive improvements in knowledge or skills attainment. Control group study results ranged from large positive (1.56) to negative effects (-0.51). Average MERSQI scores of 12.1 (range, 8.5-15.5) were comparable to GME research norms. Varying methods for implementing and studying the FC in GME has led to variable results. While residents expressed a positive attitude toward FC learning, shortcomings were reported. Approximately half of the studies comparing the flipped to the traditional classroom reported better achievement under the FC design. As indicated by the MERSQI score, studies captured by this review, on average, were as rigorous as typical research on residency education.

This systematic review aimed to identify, appraise, and synthesize evidence for the effectiveness of simulation modalities in promoting nursing students' empathy. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis was followed with 20 studies included. The Medical Education Research Study Quality Instrument (MERSQI) was used to evaluate the quality of the publications. The mean MERSQI score was 10.95 with an SD of 1.62, which was above the average in the literature.Five simulation modalities were identified: standardized patient, simulated suit, manikin, virtual simulation, and virtual reality simulation. Most of the studies (n = 15) reported significant differences in the measured outcomes after their simulation activities, including all 5 simulation modalities, but the most effective modality was the simulation suit. The simulation suit replicates the lived experiences of others, which enhances perspective taking by "turning the student into the patient" through the physical and sensory effects of simulated suits. However, outcome measures were limited to self-report instruments in the studies.

Evidence-based practice in education requires high-quality evidence, and many in the medical education community have called for an improvement in the methodological quality of education research. Our aim was to use a valid measure of medical education research quality to highlight the methodological quality of research publications and provide an overview of the recent internal medicine (IM) residency literature. We searched MEDLINE and PreMEDLINE to identify English-language articles published in the United States and Canada between January 1, 2010, and December 31, 2011, focusing on IM residency education. Study quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI), which has demonstrated reliability and validity. Qualitative articles were excluded. Articles were ranked by quality score, and the top 25% were examined for common themes, and 2 articles within each theme were selected for in-depth presentation. The search identified 731 abstracts of which 223 articles met our inclusion criteria. The mean (±SD) MERSQI score of the 223 studies included in the review was 11.07 (±2.48). Quality scores were highest for data analysis (2.70) and lowest for study design (1.41) and validity (1.29). The themes identified included resident well-being, duty hours and resident workload, career decisions and gender, simulation medicine, and patient-centered outcomes. Our review provides an overview of the IM medical education literature for 2010-2011, highlighting 5 themes of interest to the medical education community. Study design and validity are 2 areas where improvements in methodological quality are needed, and authors should consider these when designing research protocols.

BackgroundSurgeons constantly deal with high-demand tasks in the operating room that require simultaneously processing a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on surgeons’ performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons’ cognitive load, and a critique of the utility, reliability and validity of current assessment metrics.MethodsA search strategy encompassing MEDLINE, EMBASE, Web of Science, PsycINFO, ACM-DL, IEEEXplore, PROSPERO and Cochrane was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe: study design, setting, specialty, participants, cognitive load measures and MERSQI score.ResultsOf 391 articles retrieved, 84 met inclusion criteria, totaling 2,053 unique participants. Most studies were carried out in simulated settings (59 studies, 70%). Sixty-one studies (73%) used self-report methods, of which NASA-TLX was the most applied tool (44 studies, 52%). Heart rate variability analysis was the most used real-time method (11 studies, 13%).ConclusionSelf-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical modalities and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, since they allow capture of cognitive load fluctuations. A combination of both subjective and objective methods might provide an enhanced measuring of surgeons’ cognition.

Effectiveness of digital resuscitation training in improving knowledge and skills: A systematic review and meta-analysis of randomised controlled trials

Integrated or collaborative care is a well-evidenced and widely practiced approach to improve access to high-quality mental health care in primary care and other settings. Psychiatrists require preparation for this emerging type of practice, and such training is now mandatory for Canadian psychiatry residents. However, it is not known how best to mount such training, and in the absence of such knowledge, the quality of training across Canada has suffered. To guide integrated care education nationally, we conducted a systematic review of published and unpublished training programs. We searched journal databases and web-based 'grey' literature and contacted all North American psychiatry residency programs known to provide integrated care training. We included educational interventions targeting practicing psychiatrists or psychiatry residents as learners. We critically appraised literature using the Medical Education Research Study Quality Instrument (MERSQI). We described the goals, content, and format of training, as well as outcomes categorized according to Kirkpatrick level of impact. We included 9 published and 5 unpublished educational interventions. Studies were of low to moderate quality and reflected possible publication bias toward favourable outcomes. Programs commonly involved longitudinal clinical experiences for residents, mentoring networks for practicing physicians, or brief didactic experiences and were rarely oriented toward the most empirically supported models of integrated care. Implementation challenges were widespread. Similar to integrated care clinical interventions, integrated care training is important yet difficult to achieve. Educational initiatives could benefit from faculty development, quality improvement to synergistically improve care and training, and stronger evaluation. Systematic review registration number: PROSPERO 2014:CRD42014010295.

BackgroundDespite a recent rise in publications describing extracorporeal membrane oxygenation (ECMO) education, the scope and quality of ECMO educational research and curricular assessments have not previously been evaluated.ObjectiveThe purposes of this study are 1) to categorize published ECMO educational scholarship according to Bloom’s educational domains, learner groups, and content delivery methods; 2) to assess ECMO educational scholarship quality; and 3) to identify areas of focus for future curricular development and educational research.MethodsA multidisciplinary research team conducted a scoping review of ECMO literature published between January 2009 and October 2021 using established frameworks. The Medical Education Research Study Quality Instrument (MERSQI) was applied to assess quality.ResultsA total of 1,028 references were retrieved; 36 were selected for review. ECMO education studies frequently targeted the cognitive domain (78%), with 17% of studies targeting the psychomotor domain alone and 33% of studies targeting combinations of the cognitive, psychomotor, and affective domains. Thirty-three studies qualified for MERSQI scoring, with a median score of 11 (interquartile range, 4; possible range, 5–18). Simulation-based training was used in 97%, with 50% of studies targeting physicians and one other discipline.ConclusionECMO education frequently incorporates simulation and spans all domains of Bloom’s taxonomy. Overall, MERSQI scores for ECMO education studies are similar to those for other simulation-based medical education studies. However, developing assessment tools with multisource validity evidence and conducting multienvironment studies would strengthen future work. The creation of a collaborative ECMO educational network would increase standardization and reproducibility in ECMO training, ultimately improving patient outcomes.

Ambulatory training in internal medicine residency programs has historically been considered less robust than inpatient-focused training, which prompted a 2009 revision of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements in Internal Medicine. This revision was intended to create a balance between inpatient and outpatient training standards and to spur innovation in the ambulatory setting. We explored innovations in ambulatory education in internal medicine residency programs since the 2009 revision of the ACGME Program Requirements in Internal Medicine. The authors conducted a scoping review of the literature from 2008 to 2017, searching PubMed, ERIC, and Scopus databases. Articles related to improving educational quality of ambulatory components of US-based internal medicine residency programs were eligible for inclusion. Articles were screened for relevance and theme categorization and then divided into 6 themes: clinic redesign, curriculum development, evaluating resident practice/performance, teaching methods, program evaluation, and faculty development. Once a theme was assigned, data extraction and quality assessment using the Medical Education Research Study Quality Instrument (MERSQI) score were completed. A total of 967 potentially relevant articles were discovered; of those, 182 were deemed relevant and underwent full review. Most articles fell into curriculum development and clinic redesign themes. The majority of included studies were from a single institution, used nonstandardized tools, and assessed outcomes at the satisfaction or knowledge/attitude/skills levels. Few studies showed behavioral changes or patient-level outcomes. While a rich diversity of educational innovations have occurred since the 2009 revision of the ACGME Program Requirements in Internal Medicine, there is a significant need for multi-institution studies and higher-level assessment.