Analyzing and Interpreting Data From Likert-Type Scales

Editor’s note: The opinions expressed in this editorial do not necessarily reflect the opinions of the AAMC or its members. This issue of Academic Medicine is devoted to an exciting set of research reports, reviews, and commentaries selected for publication from more than 60 submissions as part of our ongoing collaboration with the Research In Medical Education (RIME) program.1–17 The topics are wide-ranging—experiences of moral distress among trainees, interprofessional team leadership, assessment and feedback during training, and the role of arts and humanities in professional identity formation, to name just a few. As in the past, each of these published articles will be presented and discussed as part of the RIME program at Learn Serve Lead 2024: The AAMC Annual Meeting, which takes place this month. RIME’s work highlights creativity and quality in education scholarship and communicates the importance of medical education inquiry to stakeholders throughout academic medicine, such as institutional leaders, promotions committee members, and learners. RIME articles elevate issues of concern to medical education in a timely manner. Such attention can lead to dialogue, consensus, and advances in the field. RIME also supports the professional development of medical education scholars. Manuscripts submitted for consideration by the RIME Program Planning Committee receive written feedback from reviewers and committee members, and service on the RIME committee itself is a valuable learning experience and opportunity for leadership in medical education and academic medicine. In celebrating RIME’s 60th anniversary in 2021, members of the RIME Program Planning Committee conducted a survey of past RIME chairs to better understand how RIME has helped advance medical education research.18 The past chairs recognized RIME as a catalyst for medical education research, strengthening the rigor and impact of medical education research, developing a community of scholars, and supporting academic career development. They also identified as priorities the need for greater innovation and translation of medical education research to practice. They welcomed new activities and collaborations, such as faculty mentorship initiatives and support for educators pursuing advanced degrees in education.18 Each year leaders on the RIME committee prepare a commentary, published as part of the RIME collection, that addresses a way to improve medical education research in academic medicine. In 2012, RIME leaders called for a more intentional, strategic, and inclusive effort. Medical education research requires an intellectual structure, a framework into which seemingly random studies can come together to improve learning and practice. The challenge before us is to look forward and develop a well-laid plan to further the field of medical education research nationally and internationally.19 For this year’s commentary, Sukhera and colleagues1 similarly challenge the field of academic medicine to do work that will improve learning and practice. They encourage medical education researchers to think beyond rigor, relevance, and utility in their work and to find ways to generate new knowledge that is readily—and will actually be—used. Drawing upon insights from implementation science, the authors call upon medical education researchers to mobilize toward actionable, scalable, impactful scholarship. Over the years, Academic Medicine has partnered with RIME to publish 30 journal supplements; RIME collections have also appeared in our regular issues in October 2012, October 2013, and November 2014. The editorial teams of Academic Medicine and our AAMC sister journal, MedEdPORTAL, have also been fortunate to benefit from the service of members of the RIME community, such as manuscript authors, creators of curricular and education resources, and expert reviewers. Authors of RIME articles have been featured on the Academic Medicine Podcast, which further increases the reach of the scholarly contributions. Academic Medicine also was pleased to collaborate with the leadership of RIME in 2022 when it launched its mentorship program for early and underrepresented faculty scholars, a program that has evolved to become MedEdSCHOLAR.20 This year, we are pleased to feature the RIME articles and commentaries in this issue of Academic Medicine. This decision reflects our continued strong support for the work of the RIME planning committee, RIME authors, and the broader RIME community in furthering medical education scholarship.

Medical education research in general is a young scientific discipline which is still finding its own position in the scientific range. It is rooted in both the biomedical sciences and the social sciences, each with their own scientific language. A more unique feature of medical education (and assessment) research is that it has to be both locally and internationally relevant. This is not always easy and sometimes leads to purely ideographic descriptions of an assessment procedure with insufficient general lessons or generalised scientific knowledge being generated or vice versa. For medical educational research, a plethora of methodologies is available to cater to many different research questions. This article contains consensus positions and suggestions on various elements of medical education (assessment) research. Overarching is the position that without a good theoretical underpinning and good knowledge of the existing literature, good research and sound conclusions are impossible to produce, and that there is no inherently superior methodology, but that the best methodology is the one most suited to answer the research question unambiguously. Although the positions should not be perceived as dogmas, they should be taken as very serious recommendations. Topics covered are: types of research, theoretical frameworks, designs and methodologies, instrument properties or psychometrics, costs/acceptability, ethics, infrastructure and support.

The field of graduate medical education (GME) research is attracting increased attention and broader participation. The authors review the special ethical and methodological considerations pertaining to medical education research. Because residents are at once a convenient and captive study population, a risk of coercion exists, making the provision of consent important. The role of the institutional review board (IRB) is often difficult to discern because GME activities can have multiple simultaneous purposes, educational activities may go forward with or without a research component, and the subjects of educational research studies are not patients. The authors provide a road map for researchers with regard to research oversight by the IRB and also address issues related to research quality. The matters of whether educational research studies should have educational value for the study subject and whether to use individual information obtained when residents participate as research subjects are explored.

How and why social science theory can contribute to medical education research

Medical education researchers are inherently collaborators. This paper presents a discussion of theoretical frameworks, issues and challenges around collaborative research to prepare medical education researchers to enter into successful collaborations. It gives emphasis to the conceptual issues associated with collaborative research and applies these to medical education research. Although not a systematic literature review, the paper provides a rich discussion of issues which medical education researchers might consider when undertaking collaborative studies. Building on the work of others, we have classified collaborative research in three dimensions according to: the number of administrative units represented; the number of academic fields present, and the manner in which knowledge is created. Although some literature on collaboration focuses on the more traditional positivist perspective and emphasises outcomes, other literature comes from the constructivist framework, in which research is not driven by hypotheses and the approaches emphasised, but by the interaction between investigator and subject. Collaborations are more effective when participants overtly clarify their motivations, values, definitions of appropriate data and accepted methodologies. These should be agreed upon prior to commencing a study. The way we currently educate researchers should be restructured if we want them to be able to undertake interdisciplinary research. Despite calls for researchers to be educated differently, most training programmes for developing researchers have demonstrated a limited, if not contrary, response to these calls. Collaborative research in medical education should be driven by the problem being investigated, by the new knowledge gained and by the interpersonal interactions that may be achieved. Success rests on recognising that many of the research problems we, as medical educators, address are fundamentally interdisciplinary in nature. This represents a transition to bridge the dichotomy often presented in medical education between theory building and addressing practical needs.

Medical education research is underfunded. Little research focuses on basic issues of learning and teaching in medicine, and few educational innovations are tested across institutions to insure their generalizability. Despite repeated calls for the development of research organizations that would deal with substantial issues in medical education, funding remains a miniscule portion of the total budget for medical education. The author discusses how medical education research is supported in the present economic environment. The majority of published research is accomplished during the course of the researcher's employment. Many universities support pilot studies that lead to larger grants. Faculty development programs and career or academic awards support some research. Small to moderate amounts of funding are available through a variety of sources including medical specialty associations and private foundations. The author lists a variety of resources and strategies for finding foundations to fund research. Funding for larger research projects, particularly those that require more than a few years to accomplish or that involve multiple institutions, is more difficult to obtain. Federal training grants provide the laboratory for innovation and education research, but include inadequate funds to accomplish in-depth evaluation or research. The lack of emphasis on theory building and failure to prove the usefulness of innovations across schools cannot be overcome with current levels of funding. Medical educators must be advocates for the research support needed to move medical education research from a field with limited influence on practice to one that is able to connect sound insights and ideas with actual practices in education and improved patient health.

ContextLongitudinal qualitative research is an approach to research that entails generating qualitative data with the same participants over extended periods of time to understand their lived experiences as those experiences unfold. Knowing about dynamic lived experiences in medical education, that is, learning journeys with stops and starts, detours, transitions and reversals, enriches understanding of events and accomplishments along the way. The purpose of this paper is to create access points to longitudinal qualitative research in support of increasing its use in medical education.MethodsThe authors explore and argue for different conceptualisations of time: analysing lived experiences through time versus analysing lived experiences cross‐sectional or via 2‐point follow‐up studies and considering time as subjective and fluid as well as objective and fixed. They introduce applications of longitudinal qualitative research from several academic domains: investigating development and formal education; building longitudinal research relationship; and exploring interconnections between individual journeys and social structures. They provide an illustrative overview of longitudinal qualitative research in medical education, and end with practical advice, or pearls, for medical education investigators interested in using this research approach: collecting data recursively; analysing longitudinal data in three strands; addressing mutual reflexivity; using theory to illuminate time; and making a long‐term commitment to longitudinal qualitative research.ConclusionsLongitudinal qualitative research stretches investigators to think differently about time and undertake more complex analyses to understand dynamic lived experiences. Research in medical education will likely be impoverished if the focus remains on time as fixed. Seeing things qualitatively through time, where time is fluid and the past, present and future interpenetrate, produces a rich understanding that can move the field forward.

BackgroundMedical education research plays a critical role in improving patient care and advancing medical practices, yet the engagement among healthcare professionals remains limited. This study aims to explore the factors that influence participation in medical education research and provide recommendations to foster a research-inclusive culture within healthcare institutions.MethodsA comprehensive literature review was conducted across four databases: Web of Science, MEDLINE, Scopus, and Airiti Library. The search focused on studies published between 2002 and 2024, utilizing keywords related to ‘medical education research,’ facilitate,’ ‘engage,’ and ‘involve’. The initial search yielded 814 records. After removing duplicates and conducting a thorough screening process based on predefined inclusion and exclusion criteria, eight studies were selected for in-depth analysis. These studies were evaluated to identify key facilitators and barriers that influence research participation among healthcare professionals.ResultsThe analysis, guided by the Ecological Model, revealed a complex interplay of facilitators and barriers affecting research engagement. At the individual level, opportunities for professional development and career advancement emerged as significant facilitators. The interpersonal level highlighted the importance of leadership, mentorship, and peer support in fostering engagement. The environmental level, including institutional policies, funding availability, and protected research time, were identified as critical influences on participation.ConclusionsThe findings suggest that a holistic approach is needed to promote participation in medical education research, addressing factors at the individual, interpersonal and environmental levels. Recommendations include developing targeted strategies to improve intrinsic motivation among healthcare professionals, strengthening mentorship and peer support programs, and implementing supportive institutional policies that allocate dedicated research time and secure financial resources. By fostering a culture that values and facilitates medical education research, healthcare institutions can drive innovation in medical education, ultimately enhancing the quality of patient care and safety.

Evaluation of medical education is a very complex and challenging process. The Supreme Court of the United States, in trying to define when free speech becomes pornography, noted simply that when you see it, you will know it. The proverbial blind men trying to define the elephant defined the pachyderm by describing the individual parts, without being able to se the whole picture. Miller, in his 1989 invited address1 to the Research in Medical Education group, part of the Association of American Medical Colleges, suggested a pyramid as the framework for assessment of clinical performance, the bottom representing Knowledge and the top Action. Most of the clinical assessment on which we rely currently is at the bottom of the pyramid; ie, knowledge. We use in-house testing as a way to examine resident cognitive skills, and the American Board of Pediatrics continues that approach with a written examination for certification and recertification. We pay lip service to advancing up the pyramidal model, with only some programs using standardized patients, other simulations, and directly observing resident performance in a disciplined manner. Computer technology and other resources offer an opportunity to change the way we evaluate trainees and assure the public that we are graduating competent, caring pediatricians.Evaluation should be not only summative, but must be formative as well. We always need to ask the question, “How can I improve my skills”? As educators, we should ask, “How can we improve the skills of the learners?” In addition, educational evaluation should consider assessment of other elements of the educational enterprise: the curriculum, faculty, record system, clinical experience, feedback system, self-assessment process, interpersonal and educational skills, psychosocial and humanistic skills, as well as ethical approaches to patient care, just to name a few. Thus, like the proverbial elephant, not only is evaluation a massive undertaking, it is also highly complex in nature. No single method of evaluation will suffice. Multiple approaches must be used, with often minimal to no correlation among the strategies. Two excellent reviews summarize the multiple perspectives associated with the evaluation of residency competence.2,3Research in medical education has improved greatly in the past few years. Recent studies rely on increasingly objective methods of measuring clinical competency. Control and comparison groups are more frequently used to determine if the intervention directly affects performance. Qualitative approaches offer additional ways to assess residency education and program evaluation. Newer methods of objective strategies, such as the OSCE (Objective Structural Clinical Examination), have been introduced, with some evidence of efficacy, but with remaining questions of whether this method can be the gold standard or just another way of assessing the complex nature of clinical skills.4,5 The standardized patient program, including the use of children as standardized patients, has been effective in evaluating teaching clinical skills, although some colleagues call these patients fake.6–8In this issue of the Journal of the Ambulatory Pediatric Association, Lopreiato and colleagues9 present an excellent example of an evaluation of a residency program, assessing curriculum, knowledge, and clinical skills, in an important component of pediatrician competence: health maintenance. They answered the important question of educational research: “What difference did the intervention make?” This research requires control or comparison groups, although it is often difficult to have reasonable comparison groups. Lopreiato chose an historic contemporaneous comparison group from his own institution, so as to control variability of the educational context as much as possible. He also used standardized patient mothers to assess clinical skills, real patients who bring true life experiences as mothers seeking medical care for their children. Mothers were trained to give effective feedback and evaluation to residents, while not having scripts. Their method, in our view, is important, combining reliable and valid ways to measure clinical skills. It is also a creative attempt to standardize the evaluation process, yet keep it as real as possible for the learners. They also assessed the curriculum, used multiple sources for evaluation (including a systematic chart audit), and documented improvement in clinical skills. Their methodology allows them to assess those areas with no improvement of skills, so that they can plan for future improvement. For this exemplary work, Lopreiato and colleagues received the Ray E. Helfer Award for Innovation in Pediatric Education from the Pediatric Academic Societies for 1998. Congratulations!

Medical education research faces increasing pressure to demonstrate impact and utility. These pressures arise amidst a climate of accountability and within a culture of outcome measurement. Conventional metrics for assessing research impact such as citation analysis have been adopted in medical education, despite researchers' assertion that these quantitative measures insufficiently reflect the value of their work. Every knowledge community has its own definitions of what counts as knowledge, how that knowledge should be produced, and how the quality of that knowledge production should be evaluated. Definitions of impact and knowledge shape and constrain researchers' foci and endeavors. Therefore, metrics that meaningfully evaluate the knowledge outputs of researchers need to be defined within each field. It is time for medical education research, as a field, to examine how to measure research impact and carefully consider the broader implications these measures may have. The authors discuss developments in research metrics more broadly, then critically examine impact metrics currently used in the medical education field and propose alternatives to more meaningfully track and represent impact in medical education research. Grey metrics and narrative impact stories to more fully capture the richness and nuanced nature of impact in medical education research are introduced. The authors advocate for a continual examination of how impact is defined, eschewing unquestioned use of conventional metrics. A new conversation is needed, as well as a research agenda to help medical education conceptualize and study metrics more appropriate for the field.

The development of successful educational scholarship, either curricula or medical education research, is vital in ensuring that the field of medical education continues to evolve. Fostering the skills of medical educators in conducting high-quality educational research is essential to this process because publishing such research helps to disseminate best educational practices to the medical community at large. Unfortunately, developing rigorous medical education research can be challenging for pediatric hospitalists within busy clinical settings. In this article, we aim to discuss key principles and frameworks for curricular development as well as offer guidance in transforming a curriculum into a scholarly medical education research product for pediatric hospital medicine providers.

Medical education is a rapidly growing field of research, incorporating diverse disciplinary perspectives to assist physician trainees in developing the complex skills needed for practice. Education science is happening in many medical specialties; however, Family Medicine or General Practice settings have not seen a proportional share of theory-driven education research. The limited nature of education research in Family Medicine is surprising, given that there are several aspects of general practice that make it a particularly unique and interesting context to study issues of general importance to medical education, and there is a particular need for education research to further the discipline of Family Medicine. It is important that the community of medical education researchers in Family Medicine have a strong understanding and perspective on the breadth and potential impact of their work, and what this means for the training that occurs within and for the discipline. This Reflection aims to inform strategic thinking, collaboration, and innovation in medical education research as it pertains to Family Medicine. It does so by discussing four hallmarks of Family Medicine practice and outlining their independent and interactive potential for medical education research.

Directors of offices of research in medical education were surveyed in July of 1983 in order to assess changes within their offices occurring since a similar survey conducted in 1979. The goals, organizational relationships, effort devoted to various educationally-related activities, and current research endeavors were obtained from 39 offices. Descriptive statistics are presented which summarize the responses to the survey, including a detailed content analysis of 175 reported active research projects. Comparison data for 24 offices of research in medical education participating in both the 1979 and 1983 studies are also presented. Findings indicate that despite only modest increases in budget since 1979, offices have expanded both their research efforts as well as their involvement in faculty development. Other trends are discussed and compared to the findings of a survey of instructional improvement centers in higher education.

Background: Since the 1970s, the Dutch have been active innovators and researchers in the medical education domain. With regards to the quantity of publications in the medical education literature, the Netherlands rank second among countries in Europe and fourth worldwide over the past years, related to the journals with highest impact factors.Aim: We attempted to analyse what made this country so productive by exploring the backgrounds of this success.Method: An updated comparative evaluation was conducted. Nationalities of first and last authors were screened. And an indicator whether supervision is ‘exported’ or ‘imported’ was calculated.Results: The Netherlands still rank high in number of publications and the production increases. In contrast to other countries, the Dutch are ‘exporting’ their supervision.Conclusions: The opportunity to start a new, experimental medical school from scratch in Maastricht has undoubtedly contributed significantly to this national productivity. One other factor seems the establishment of the Netherlands Association for Medical Education with its mission to stimulate research and development of education. Annual conferences, Courses on research in medical education, Chairs in medical education qualified to graduate PhD students, and a general open and critical national culture of enquiry may have added to this success.

Objective and Aim The growing emphasis on accountability, quality assurance, and outcomes-based education has intensified the demand for robust and critically appraised evidence in medical education. While the principles of evidence-based medicine have substantially shaped clinical decision-making, their direct transfer to educational contexts has proven conceptually and methodologically challenging. Objective: This systematic review aims to synthesize the literature on evidence types, critical appraisal frameworks, and methodological quality standards in medical education research, and to present an integrative appraisal matrix suitable for educational decision-making. Methods A systematic review was conducted in accordance with PRISMA 2020 guidelines (7). Searches were performed in MEDLINE (PubMed), ERIC, Scopus, and Web of Science for studies published between 2000 and 2024. Included studies addressed evidence hierarchies, critical appraisal, research quality instruments, or methodological frameworks in medical or health professions education. Data were synthesized narratively due to methodological heterogeneity. Results Eighty-four studies met inclusion criteria. Evidence in medical education was categorized into quantitative, qualitative, mixed-methods, and design-based research. Traditional biomedical hierarchies were insufficient to capture educational relevance and contextual complexity. Appraisal tools such as BEME, MERSQI, CASP, and realist review frameworks were commonly cited, yet inconsistently applied. Conclusions Critical appraisal in medical education requires a pluralistic, theory-informed, and context-sensitive approach. The proposed appraisal matrix integrates methodological rigor, educational relevance, outcomes meaningfulness, and transferability to support informed educational practice. Keywords: Medical education research; Systematic review; Critical appraisal; Evidence hierarchy; Research methodology