Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

On July 1, 2013, the first group of programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) transitioned into the Next Accreditation System (NAS), and 1 year later, the remaining accredited specialties, the transitional year, and the review of sponsoring institutions will move into the NAS.1 The NAS is an outcomes-based and improvement-focused approach to the accreditation of programs and sponsoring institutions. The intent is to both ensure a level of quality (through assessment of compliance with the accreditation standards) and to promote ongoing improvement in all programs and innovation in high-performing programs.1 The added focus on improvement will be achieved through (1) a Self Study that each program will complete before its 10-year site visit and includes aggregation and analysis of data on its longitudinal improvement effort, and (2) a Self Study site visit during which the program will describe its improvement process and the outcomes achieved. The Self Study site visit will occur approximately 1 year after the program begins its Self Study.The lengthening of site visit intervals in the NAS and the focus on outcomes and continuous improvement call for a new type of site visit, and this new model requires testing. To further inform the Self Study and to increase its relevance to programs, the ACGME envisioned a new focus on programs' aims, combined with an environmental assessment that focuses on strengths, areas for improvement, opportunities, and threats.A key aim in the NAS is to reduce the burden of accreditation, and a recognized source of burden is the preparation of site visit documentation. Thus, the Self Study site visit also calls for a new approach to reporting that provides insight into programs' longitudinal improvement efforts, without that documentation either adding undue burden or affecting the ability of the program to conduct a complete and honest accounting of its strengths and its areas in need of improvement.The Self Study approach is currently being refined, with the first Self Studies for programs in Phase I of the NAS scheduled to begin in 2014, and the first Self Study site visits planned for 2015. In this brief article, we offer early information from the ongoing, conceptual testing of the Self Study site visit.By 2012, the aims and content of the 10-year Self Study and Self Study site visit (box 1) had evolved sufficiently to allow the ACGME to test the new elements of the proposed approach. This included the enhanced focus on program improvement, using the Annual Program Evaluation, and the new focus on program aims and an assessment of programs' performance and opportunities and threats in their local environment.In appreciation of the significant development the Self Study process and the new type of site visit would require, ACGME leadership thought it would be beneficial to test the emerging concepts and approaches in a sample of Self Study test visits, beginning in late 2012. It was clear that a “real” Self Study site visit would have required the programs to have completed a self-assessment and that the test visits would test only selected elements of the planned approach.To date, site visits at 3 institutions involving internal medicine and plastic and vascular surgery core programs and subspecialty programs in those areas have been completed. Test visits involved volunteer institutions, and the findings were not shared with Residency Review Committees. All test site visit programs were visited by a team, as is planned for the actual Self Study site visits. The individuals interviewed and the time commitment for participants on the day of the site visit were comparable to a “regular” site visit. The ACGME sought to ensure that preparation before a site visit would be minimal, and the site visit was limited to asking the residents or fellows in the test programs to complete a consensus list of program strengths and areas for improvement.The test visit entailed a debriefing with site visitors and program participants and produced reports for internal sharing and comprehensive field notes. In contrast to the more general compliance focus that had characterized earlier tests of Program Information Form-less site visits,2 these visits tested selected new elements planned for future Self Study visits. They included a focus onItems 2 through 4 above constituted an initial test of new dimensions the ACGME envisioned as part of a future NAS 10-year Self Study site visit. One of the key information items for this portion of the visit was the Annual Program Evaluation. To continue to preserve the ability of programs to conduct a frank assessment of areas for improvement, the test visits maintained the confidentiality of the Annual Program Evaluation, and the team asked the program director to discuss program improvement, articulating successes achieved and areas still being worked on, based on the several recent Annual Program Evaluations. The ACGME envisions that a similar approach will be used for actual Self Study visits.Although the sample is small, the test site visits to date have confirmed the feasibility and “assess-ability” of the added dimensions of a Self Study visit. Programs articulated “aims” without difficulty, although, other than efforts to address lower than desirable board performance, offering general information on preparing residents for academic careers, and in 1 instance, starting a primary care track, respondents were less clear about their ongoing activities to enhance performance relevant to these aims.Discussion during the assessment of strengths, areas for improvement, opportunities, and threats focused primarily on national threats, particularly changes in federal support for graduate medical education (GME). Local opportunities and threats were less readily identified in the conversations with program leadership. This suggests that these new site visit dimensions would benefit from ongoing assessment over time, and it is expected that these areas would be more developed in a “real” Self Study, which would have been preceded by a formal Self Study process.The early test site visits also highlighted several areas of emerging focus, some likely prompted by the increased emphasis on educational outcomes in the NAS. During the site visits, programs discussed new modes of evaluation on the Milestones; on initiatives to restructure their didactic programs, focusing on scheduling and delivery models to maximize access and participation; and on measuring the effect of the didactic offerings. Preparation for the board examination, as the most visible element of outcomes focus in the current system, constituted a major focus of the discussion of educational outcomes across programs at the programs sites visited to date. Another focal area likely resulted from new roles for faculty in the NAS, particularly professional development for assessing performance on the milestones and participation on Clinical Competency Committees. In addition to faculty development content, discussions addressed approaches to encourage participation, including scheduling strategies and financial and other incentives, such as academic promotion and recognition.Finally, the visits highlighted some beneficial approaches for tracking data on longitudinal improvement, using simple, common institutional formats that allowed programs to record and report their improvements and areas still being worked on. This longitudinal tracking of improvement will gain in importance in the NAS, as much of the focus of the Self Study visit will be on programs' effectiveness in making ongoing improvements.The early tests contributed significantly to the development and refinements of the program Self Study approach and the associated site visits envisioned for the NAS. This work will continue with a focus on (1) offering guidance to programs about effective and efficient approaches for longitudinal improvement, including sessions at the ACGME Annual Educational Conference; (2) developing a report that aggregates NAS performance data the ACGME would provide to programs at the “kick-off” of their Self Study effort, approximately 1 year before the scheduled site visit; and (3) formulating a succinct document for programs to complete in preparation for a Self Study site visit.During the early test visits, it became clear that a truly meaningful Self Study visit would require the prior completion of a Self Study process. The concept is widely used in educational accreditation, yet it remains one of the more debated elements. The ideal approach for an NAS Self Study would produce a self-assessment that aids in the acquisition of knowledge about the program and identifies areas of real, tangible improvements.3 At the same time, organizational and programmatic Self Studies, although valuable from an improvement perspective, carry significant time, financial, and opportunity costs.4 It is clear that, given the smaller size of residency programs or even a core programs and its associated subspecialty programs compared with universities or medical schools and the resource constraints in the GME environment, the Self Study approach will need to be adapted and streamlined for use in the NAS. It also became apparent that the process would need to address scheduling challenges for programs, particularly when core programs and their subspecialty programs are seen as a group, as is planned in the NAS. The rationale and anticipated benefits for having a collective Self Study and Self Study site visit for core programs and their associated subspecialty programs are shown in box 2.The exploration of elements of the Self Study and the associated visit confirmed the feasibility of the approach and acceptance by program and institutional participants, although the sample was limited and the use of the information for internal improvement may have produced a more receptive and less critical audience. Future challenges and areas to further develop and refine the site visit approach in the NAS will need to deal with efforts to assess programs' ongoing focus on improvement while ensuring the confidentiality of the data underlying that process.Additional site visits are planned, with a particular focus on tapping into programs' ongoing improvement efforts, and how those efforts could be documented without impinging on their ability to conduct frank and forthright assessments of strengths and areas for improvement. Future work will also explore formats that offer added value to programs, such as a Self Study or site visit that yields valuable insight for programs in key areas, particularly recognition of opportunities and threats within their environment.

Implementing the Next Accreditation System: Results of the 2014-2015 Annual Data Review.

The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project has shifted the focus for accrediting residency training programs from structure and process to documented outcomes. A central part of the ACGME's Next Accreditation System (NAS) is “articulating milestones of competency development in each discipline.”1 Resident achievement of educational Milestones will be an indicator of the educational effectiveness of residency programs. Most specialties have developed their Milestones through a collaborative effort involving representatives from the ACGME, the pertinent American Board of Medical Specialties (ABMS) organization, the pertinent professional specialty society, and the pertinent program directors' association. In contrast, the approach was quite different for Transitional Year (TY) residency programs, which comprise only 1 year of accredited postgraduate medical education, where there is no corresponding ABMS board certification entity or professional specialty society. In addition, graduates of TY training programs are not expected to demonstrate sufficient competency to enter the independent practice of medicine, as is the case with other specialty (categorical) residency programs.

The Accreditation and Certification System After Next

Before exploring the projected future of Sponsoring Institutions (SIs), it is important to understand their origins and development. Graduate medical education (GME) has relied on diverse learning communities to educate residents for unsupervised practice. As 1 such learning community, the SI has contributed to the education of residents and fellows by ensuring the provision of support systems, resources, and administrative structures. Through their oversight of GME, SIs have fostered clinical learning and working environments in which residents and fellows achieve educational milestones that indicate their ability to provide high-quality, safe patient care upon completion of their programs and throughout their professional lives. What follows is a brief history of institutional accreditation by the Accreditation Council for Graduate Medical Education (ACGME), the process by which we have come to recognize the SI as that community necessary to form physicians.In 1982, the newly formed ACGME published its “Essentials of Accredited Residencies in Graduate Medical Education,” which included a set of “General Requirements” that established standards and responsibilities for GME programs. Part I of these General Requirements began with the declaration that “[p]rograms in graduate medical education are sponsored by institutions engaged in providing medical care and health services. The principal institutions for graduate medical education are hospitals.”2 In Section 1, “Responsibility of Institutions,” the ACGME outlined standards that remain at the core of the essential relationships and processes expected of every institution that seeks to educate residents: commitment to sponsorship agreed upon by both faculty and administration; distribution of resources for educational purposes; establishment of institutional policies by which program directors exercised various responsibilities for their respective programs; periodic analyses of each program's effectiveness in meeting its goals and objectives; provision of facilities and resources to support education; and maintenance of hospital accreditation.3 It is important to note that, at the time, these standards were not new to GME; they were based on well-established groundwork laid by the ACGME's predecessors in program accreditation, namely, the American College of Surgeons, the American Board of Internal Medicine, the American Medical Association's Council on Medical Education, and the Liaison Committee on Graduate Medical Education. Each of these organizations had previously acknowledged the hospital's role as GME sponsor and had included some variation of these expectations in their respective requirements.To fulfill its accreditation mission, the ACGME delegated authority to specialty-specific Review Committees (RCs) that evaluated individual residency programs for compliance with both the General Requirements and specialty-specific program requirements. Although the ACGME's General Requirements acknowledged SIs and the role they played in GME, attention to how these institutions fulfilled their responsibilities was initially provided in the context of individual residency program review conducted by RCs. While these specialty-specific RCs made accreditation decisions independently of each other, each Committee's activity was reviewed periodically by the ACGME through its Monitoring Committee. As a result, in these earliest years of ACGME's development as an accrediting body, there was no process by which the institutions were reviewed separately from specialty programs, nor was there consideration of how a single institution demonstrated compliance with its responsibilities as specified in the General Requirements for all the programs it sponsored.Gradually, the ACGME and its RCs began to observe that noncompliance with program requirements often had, at least, some correspondence to areas of institutional responsibility identified in the General Requirements. In 1992, as a result of this increasing awareness, the ACGME approved a separate section of the General Requirements identified specifically as Institutional Requirements, along with a process for institutional review that was initially administered through the ACGME's Monitoring Committee. These Institutional Requirements elaborated on the standards that had been outlined in the original General Requirements. This initial institutional review process included a separate institutional site visit. On the basis of the site visit report and information provided in an Institutional Review Document, an institutional review administrator determined whether the SI had adequately documented and implemented policies in compliance with the General Requirements. The administrator then prepared a list of the institutions judged to be in compliance with the General Requirements. The Monitoring Committee reviewed documentation only from a representative sample of these institutions. Only 2 decision options were available. Institutions that received a favorable decision were given 5 years until their next review; institutions receiving an unfavorable decision were assigned a time frame for their next review, not to exceed 3 years. Three consecutive unfavorable decisions for an institution resulted in a loss of its ability to sponsor residency and fellowship programs.During this period, the growth of medicine was characterized in part by new specialties and subspecialties approved for certification by various specialty boards and for accreditation by the ACGME. The result was an increased workload for the RCs, due to the number of applications by academic medical centers and community teaching hospitals for new specialty and subspecialty programs. While members of RCs were expert evaluators of programs in their specialty, lengthier review agendas, coupled with the added need to understand complex issues from an institutional perspective, underscored the growing need for a separate RC to review institutions. It also became clear to the ACGME that a cadre of individuals familiar with GME from an institutional perspective, and with knowledge of the General Requirements, should provide an expert review for institutional compliance.In response to these internal and external factors in the health care delivery environment, in GME itself, and within its own internal structure, the ACGME officially constituted the Institutional Review Committee (IRC) in 1995. The IRC was structured in the same manner as the specialty-specific RCs, except with regard to appointment of its members. In contrast to the RCs, the IRC had no formal specialty board or academic medical society of individuals with leadership responsibility for GME at the institutional level from which to draw its membership. As a result, the appointing organization for the IRC became the ACGME itself, with the original members drawn from GME leadership across the country.The only options available to the newly formed IRC with regard to the review of SIs remained the previously used statuses of favorable and unfavorable. The outcomes of these reviews were not, strictly speaking, accreditation decisions, although common usage in the field often referred to them as such. Loss of the ability to sponsor accredited programs remained as the action after 3 unfavorable decisions were given to an institution. Although such a decision was never made by the IRC, it nonetheless highlighted the growing importance of institutional review. While the seeds of accreditation were sown, the time for official institutional accreditation would wait for nearly a decade.As health care delivery and funding became more complex through the 1990s, the Institutional Requirements became more detailed. Experience with the institutional review led the ACGME and the IRC to identify with greater specificity the various elements and relationships necessary for maintaining an effective SI in this environment. Managing GME relationships across the institution was an effort that assumed progressively more responsibility and importance. For example, in 1993, the standard calling for each SI to have a Graduate Medical Education Committee (GMEC) was added to the Institutional Requirements; in 1997, the appointment at each SI of a Designated Institutional Official (DIO) as chief administrator for GME was approved. The DIO and GMEC were charged with oversight of the SI's GME efforts. The DIO became the “designated” leader with whom the ACGME could communicate information that affected all programs in the SI, starting with the basic responsibility of receiving the ACGME's annual invoice for accreditation services. Eventually, the DIO became the individual in the SI who was recognized as having the authority to act on behalf of the entire ACGME-accredited enterprise. The DIO had responsibility for accreditation matters, and also assumed many locally defined duties commonly associated with management and supervisory oversight in a complex organization. The Institutional Requirements began to codify the relationships needed at the institutional level for support of the educational environment, and the formal structures that comprised the GME local learning community took shape.In the mid- to late 1990s, the ACGME underwent its first intensive strategic planning process. A central component of the resulting plan, which eventually became known as the Outcome Project, was initiated in 1999 with a grant from the Robert Wood Johnson Foundation. The central focus of this effort was the identification of 6 general competencies as organizing principles for specialty and subspecialty curricula. ACGME-accredited programs were expected to identify educational outcomes for residents and to evaluate their achievement based on the general competencies that were eventually codified into all specialty and subspecialty program requirements. During this period, the educational leadership role of the SI's DIO and GMEC became even more apparent to the ACGME, the IRC, and SIs themselves. The need for central oversight of program curricula and resident evaluation processes emerged as a critical institutional responsibility for maintaining overall educational effectiveness within each program and across the institution.After 10 years' experience observing the maturation of the institutional review process and monitoring the work of the IRC, the ACGME Board of Directors delegated full accreditation authority to the IRC in 2005. The IRC acted with this authority to accredit all SIs having 2 or more accredited programs. With this change, the IRC maintained the same structure as all RCs, with the exception of its member selection process, which remained open to nominations of individuals from the entire GME community.In 2007, the IRC completed a major revision of the Institutional Requirements in an effort to remove extraneous language and to reflect the increasing recognition of the SI's central role in overseeing GME. As further indication of the growing importance of institutional review, this revision did not occur in a vacuum. DIOs and GMECs had begun to report that areas of programs' noncompliance cited by RCs occasionally overlapped with citations at the institutional level given by the IRC. One reason for such overlap (and sometimes even dissonance or contradiction) occurred because the Institutional and Common Program Requirements had developed independently. Therefore, an ad hoc committee that included representatives from the specialty RCs and the IRC was created to reconcile these 2 sets of requirements. The result was that a major revision of the ACGME Common Program Requirements also occurred in 2007 and included a thorough reconciliation with the Institutional Requirements. This reconciliation initiative revealed how institutional accreditation gradually had come of age within the ACGME accreditation structure and at the grassroots level. Specialty-specific RCs accepted the IRC's expertise in institutional matters and were no longer compelled to cite expectations outlined in the Institutional Requirements.Consistent with the recognition of the importance of institutional review, the IRC chair eventually became a full voting member of the ACGME Council of Review Committee Chairs, thus assuring an opportunity for ongoing dialogue between the specialty RCs and the IRC. The relationship of program and institutional accreditation was forged; it mirrored the expectation of collaboration among specialty and subspecialty programs at the level of the SI, facilitated by the DIO and GMEC.Today, institutional accreditation is an integral component of the ACGME's ongoing strategic initiatives. With approval of its Next Accreditation System in 2011, the ACGME Board of Directors codified the responsibility of SIs for educational outcomes: “The ACGME accredits GME programs and SIs based on the demonstration of continuous oversight of processes and outcomes of education, and substantial compliance with accreditation standards, through the review of annually acquired information.”4 Effective institutional oversight of a single program or 100 programs is the expectation for institutional outcomes under scrutiny by the IRC as it monitors institutions on an annual basis. Recognizing this central role of the SI in monitoring outcomes, the IRC completed an extensive revision of the Institutional Requirements in 2013. This time, the revision focused on a major reordering and simplification of the language so that additional focus would be placed on demonstrable outcomes at the institutional level, even as the processes evident in the original standards remained.Yet another development in institutional accreditation occurred in 2014 when the ACGME, the American Osteopathic Association (AOA), and the American Association of Colleges of Osteopathic Medicine (AACOM) announced their agreement to form a single GME accreditation system in the United States. This decision allows graduates of allopathic and osteopathic medical schools to complete their residency and/or fellowship education in ACGME-accredited programs and to demonstrate achievement of common milestones and competencies. The move toward a single accreditation system means that many AOA-approved institutions and other entities that oversee AOA-approved programs have applied or will apply for ACGME accreditation as SIs. This landmark agreement has resulted in an expanded assortment of governing structures for SIs in which residents and fellows are educated. In addition to commonly encountered institutional models overseen by academic medical centers, community teaching hospitals, medical schools, and single-program institutions, newly ACGME-accredited SIs include osteopathic postdoctoral training institutions (OPTIs), which are educational entities that provide academic oversight and ensure GME resources for hospitals, colleges of osteopathic medicine, clinics, and teaching health sites, sometimes over large geographic areas.In 2012, the ACGME Board of Directors approved the ACGME's Clinical Learning Environment Review (CLER) program, which “provides the profession and the public a broad view of SIs' initiatives to enhance the safety of the learning environment and to determine how residents are engaged in patient safety and quality improvement activities.”5 Although not an accreditation activity, the CLER program emphasizes the institutional relationships necessary to improve health care and population health.Sponsoring Institutions now define themselves in various ways, often departing from the model of the singular hospital originally identified in the earliest standards for accreditation. What has remained consistent throughout the development of the institution's role in GME, however, is that it is the organizing force that affects GME at all levels, and that creates space for relationships that ensure the quality of resident education.Sponsoring Institutions are currently in a state of accelerated evolution in response to major changes underway in the health care system and beyond. The next stage in the maturation of SIs will necessarily benefit from reflecting on where these changes will lead. It will require the collaboration of the entire GME community to understand how SIs can meet the needs of patients by creating physicians who are prepared to practice in 2025. As in the past, it is through these relationships that future SIs will realize our future vision for GME.

Emergency Medicine Milestones: The Next Step.

In 2010, University of Kansas Medical Center internal medicine residency program leaders concluded that their competency-based curriculum and evaluation system was not sufficient to promote accurate assessment of learners' performance and needed revision to meet the requirements of the Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System (NAS). Evaluations of learners seldom referenced existing curricular goals and objectives and reflected an "everyone is exceptional, no one is satisfactory" view.The authors identified the American Board of Internal Medicine and ACGME's Developmental Milestones for Internal Medicine Residency Training as a published standard for resident development. They incorporated the milestones into templates, a format that could be modified for individual rotations. A milestones-based curriculum for each postgraduate year of training and every rotation was then created, with input from educational leaders within each division in the Department of Internal Medicine and with the support of the graduate medical education office.In this article, the authors share their implementation process, which took approximately one year, and discuss their current work to create a documentation system for direct observation of entrustable professional activities, with the aim of providing guidance to other programs challenged with developing an outcomes-based curriculum and assessment system within the time frame of the NAS.

Internal Medicine 2035: Preparing the Future Generation of Internists.

To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees. Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging. This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE. Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, β = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher. ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

MEDICAL RESIDENCY PROGRAMS are about to undergo substantial changes in the ways the physicians of tomorrow are trained. Beginning in July, the Accreditation Council for Graduate Medical Education (ACGME) will implement the Next Accreditation System to oversee training residents in 7 specialties: emergency medicine, internal medicine, neurological surgery, orthopedic surgery, pediatrics, diagnostic radiology, and urological surgery. In July 2014, the Next Accreditation System will be implemented by all remaining specialties and ultimately cover more than 9000 medical residency programs throughout the country. The aims of the new system are to “enhance the ability of the peerreview system to prepare physicians for practice in the 21st century, to accelerate the ACGME’s movement toward accreditation on the basis of educational outcomes, and to reduce the burden associated with the current structure and process-based approach” (Nasca TJ et al. N Eng J Med. doi:10 .1056/NEJMsr1200117 [published online March 15, 2012]). The ACGME was founded in 1981 when the graduate medical education environment faced 2 major stresses: variability in the quality of resident education and the emerging formalization of subspecialty education. The ACGME emphasized program structure and increased the amount and quality of teaching. But while the group believed it was successful, new stresses emerged, such as an increasingly prescriptive nature of program requirements that reduced opportunities for innovation. Acknowledging the new stresses, the ACGME pushed for the Next Accreditation System. Under this approach, each medical residency program accredited by the ACGME will have to demonstrate that its residents have the core competencies and clinical skills to provide quality patient care and the ability to respond to ongoing developments in health care delivery. Competency will be assessed in 6 core areas: patient care, medical knowledge, practice-based learning and improvement, systems-based practice, professionalism, and interpersonal skills and communication. To measure such competencies, each specialty is developing educational milestones with collaboration among American Board of Medical Specialties certifying boards, review committees, medical-specialty organizations, program-director associations, and residents. Milestones may be measured by levels of competence. For example, for practice-based learning and improvement, a resident with level 1 competency would be able to describe basic concepts in clinical practice, epidemiology, biostatistics, and clinical reasoning. The resident would also be able to categorize the study design of a research study. To achieve level 3 competency, the resident would need to be able to apply a set of critical appraisal criteria to different types of research, including synopses of original research findings, systematic reviews and meta-analyses, and clinical-practice guidelines, and to critically evaluate information from others, such as colleagues, experts, pharmaceutical representatives, and patients. Level 5 competency would find the resident independently teaching and assessing evidence-based medicine and information-mastery techniques and citing evidence supporting several common practices. Changes in resident training mean changes in the teaching of medical students as well. Carol Aschenbrener, MD, chief medical officer at the Association of American Medical Colleges, said her member institutions are working with the ACGME, but they are also adopting their own competency-based medical education systems. “Basically, the difference between competency-based education and the way we have always done it is that competency means that what the public needs, good health outcomes, is what a physician should actually be able to do, and each specialty has to decide how to achieve this,” said Aschenbrener, who has also held dean’s office positions at the University of Iowa College of Medicine in Iowa City and who served as chancellor of the University of Nebraska Medical Center in Omaha. “And we need faculty members who can help students learn how to ask questions or read a research article. It’s not about giving them a lecture but teaching them how to think.”

ACGME's Early Adaptation to the COVID-19 Pandemic: Principles and Lessons Learned.

In the Next Accreditation System, resident performance and progression assessed through educational Milestones will be used as 1 of a number of quality indicators to evaluate the educational effectiveness of residency programs. The Anesthesiology Milestones will be used to measure the success of individual anesthesiology residents in achieving expected goals of training, and the effectiveness of the residency program will in part be evaluated based on aggregated Milestone performance data.The members of the Anesthesiology Milestone Working Group were selected by the Residency Review Committee (RRC) for Anesthesiology, with input from the American Board of Anesthesiology (ABA), the Society for Education in Anesthesia, and program directors. The working group included 9 physician anesthesiologists, the executive director of the Anesthesiology RRC, and an Accreditation Council for Graduate Medical Education (ACGME) consultant. Four of the physician members were representatives from the Anesthesiology RRC (1 also served as the designated institutional official of a university-based medical center), 2 were former directors of the ABA, 2 were anesthesiology program directors, and 1 was an anesthesiology resident in the final year of training. An anesthesiologist who did not participate in the development of the Anesthesiology Milestones was appointed to the multidisciplinary expert panel that developed cross-specialty Milestones for interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice suggested for use by all specialties. Working group members were provided access to Milestone documents developed by several specialties that were a part of the initial stage in Milestone development. These models were used as templates to approach the task of developing anesthesiology-specific Milestones.The Anesthesiology Milestone Working Group held its first face-to-face meeting in February 2012, at which the ACGME consultant, who had experience working with prior Milestone working groups, provided an overview of the approach ACGME envisioned for the development of specialty-specific Milestones and shared lessons learned from Milestone development in the Phase 1 specialties. Based on the ACGME's recommendation that each Milestone should contain 5 levels of assessment (corresponding to the range from beginner to expert), initial discussions focused on determining if Milestone levels could be correlated with levels of residency training (clinical base year, clinical anesthesia years 1–3, and graduation). After recognizing that in their final form the levels would be independent of strictly time-based markers, the group made the decision that each Milestone level would represent expectations at the end of a point on the educational continuum. As a part of that continuum Level 1 Milestones were set as the expectations at the end of clinical base training. Level 2 Milestones were set to represent training expectations at the end of foundational (nonsubspecialty) clinical anesthesia training. Level 3 Milestones were set to represent the expectations for a resident at the midpoint of training after having been exposed to all subspecialty areas within anesthesiology. Level 4 Milestones were set to represent the expectations for a physician ready to practice anesthesiology independently without supervision. Level 5 Milestones were set to represent advanced characteristics of independently practicing physicians and were considered aspirational goals (a few of which might be achieved by a subgroup of exceptional residents during their training).After completion of the first draft of the Anesthesiology Milestones, the committee solicited feedback from members of the Association of Anesthesiology Core Program Directors (AACPD) and the Society for Education in Anesthesiology during their 2012 annual meetings. At the AACPD meeting feedback was provided formally through an audience response system that allowed program directors to respond to questions of relevance, importance, and measurability of each Milestone. Following these meetings, the working group edited the Milestones based upon the feedback received. This revised draft was then distributed to all program directors who were asked to provide constructive feedback through an electronic survey tool allowing confidential, individualized feedback from the program directors to the working group. The feedback was reviewed by the committee and used to make additional modifications to the Anesthesiology Milestones.The working group developed Milestones for each of the 6 ACGME competencies over the course of multiple face-to-face meetings and teleconferences. The patient care and medical knowledge Milestones, which are unique to each specialty, were developed first, and generic Milestones developed by a multispecialty panel were used as a starting point for the development of Milestones for the competencies of interpersonal and communications skills, problem-based learning and improvement, professionalism, and systems-based practice.The working group recognized the potential increase in workload that Milestone evaluation will place on program directors and Clinical Competency Committees (CCCs), and refined the number of Milestones to ensure appropriate assessment of competencies at each level of training, while not making the Milestones so granular that the evaluation process would be too burdensome. In the final version, the number of Milestones was reduced to 25. The working group believed this number provided an appropriate balance between comprehensiveness and practicality. The distribution of the Anesthesiology Milestones across the 6 competencies is shown in the table in this Introduction.While the working group sought input from the anesthesiology educational community to ensure that the Milestones were appropriate for assessing anesthesiology resident progression, the working group also recognized that the value of the proposed Milestones would benefit from an initial assessment using a pilot study in selected programs. A representative group of 20 anesthesiology programs has been selected to participate in the pilot study. The pilot will use the CCC in each program to assess the residents using the current Milestones. An evaluation tool has been developed for the program directors to provide the working group with feedback on each Milestone. The working group anticipates that the current version of the Anesthesiology Milestones will require modification over time as the specialty gains experience with this evaluation process. Once the final version of the Milestones is incorporated into training program evaluation, the RRC and other stakeholders will need to monitor and modify the Milestones based on experience so that they provide meaningful assessment of the residents, allowing anesthesiology training programs to foster excellence and optimize the educational experience for each trainee.The goal of the working group's effort was to produce Milestones that will provide program directors and CCC members with a progressive resident assessment tool that more readily identifies areas of strength and areas in need of improvement for each trainee. As Milestone data is accumulated in a department, the performance of an individual resident can be readily compared to peers and the course of progress of individual residents and residency levels in all 6 competencies domains can be evaluated. In addition, the program will have information about resident performance across the entire residency complement that will allow it to identify areas in which to focus educational efforts.The Next Accreditation System requires that accredited programs appoint CCCs to monitor residents' progress. The CCC is not new to anesthesiology training programs; the ABA has required submission of CCC reports for more than 25 years. The working group recognized that the Milestones represent a different and important approach to assessment of resident performance and progression. Although CCCs have been instrumental in assessing residents in anesthesiology programs, incorporation of Milestones into the assessment process will require modifications of the assessment process, faculty development to understand the Milestones and their implications and changes in the assessment, and communication with residents regarding performance.Based on the addition of Milestones in the assessment process, anesthesiology programs may have to modify the composition of the CCCs to ensure representation of all subspecialties and faculty involved in all levels of training. The ideal size of the CCC will likely vary by program based on the number of residents and faculty in the program. Regardless of the size of the CCC, faculty development will be the cornerstone in the transition to Milestones as a method for assessing clinical performance and progression through training. The working group recognized that as CCCs gain experience with incorporating Milestone assessments into their decisions, modifications of assessment strategies may be required. In addition, some Milestones may be difficult to assess in the traditional clinical environment because of the changing clinical environment and case distribution. Not all residents will be presented with the same clinical case experiences; therefore, a resident may not have an opportunity within the clinical environment to demonstrate some of the characteristics encompassed within the Milestones. The use of simulation, standardized patients, and other nonclinical assessments will likely grow in importance as methods for evaluating resident performance in some areas. The Milestone process will be helpful in assessing resident progression during training and the need for remediation when a resident is not progressing through a Milestone. At the same time, the Milestone process may be helpful in evaluating the effectiveness of and role for these alternative training methods in identifying areas of weakness.The adoption of Milestones in resident assessment will advance the journey toward competency-based graduate medical education that began with the Outcomes Project. Yet with this advance comes additional challenges. Once well-defined Milestones are fully integrated into residency education, it will be necessary to reconcile competency-based end points of training with the variable rate at which Level 4 Milestones will be reached by individual residents. Resolving this dilemma will provide opportunities for innovation in future training paradigms.

The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM). Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings. During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS). The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Becoming compliant with the Accreditation Council for Graduate Medical Education (ACGME) requirements for scholarly activity and remaining compliant over time requires time and attention to the development of an environment of inquiry, which is reflected in detailed documentation submitted in program applications and annual updates. Since the beginning of the next accreditation system, all ACGME programs have been required to submit evidence of scholarly activity of both residents and faculty on an annual basis. Since 2014, American Osteopathic Association-accredited programs have been able to apply for ACGME accreditation under the Single Graduate Medical Education Accreditation initiative. The Residency Program Director, Chair, Designated Institutional Official, Faculty, and coordinator need to work cohesively to ensure compliance with all program requirements, including scholarly activity in order for American Osteopathic Association-accredited programs to receive Initial ACGME Accreditation and for current ACGME-accredited programs to maintain accreditation. Fortunately, there are many ways to show the type of scholarly activity that is required for the training of surgeons. In this article, we will review the ACGME General Surgery Program Requirements and definitions of scholarly activity. We will also offer suggestions for how programs may show evidence of scholarly activity.

IntroductionOrthopaedic surgery is one of the first seven specialties that began collecting Milestone data as part of the Accreditation Council for Graduate Medical Education's Next Accreditation System (NAS) rollout. This transition from process-based advancement to outcome-based education is an opportunity to assess resident and faculty understanding of changing paradigms, and opinions about technical skill evaluation.MethodsIn a large academic orthopaedic surgery residency program, residents and faculty were anonymously surveyed. A total of 31/32 (97%) residents and 29/53 (55%) faculty responded to Likert scale assessments and provided open-ended responses. An internal end-of-rotation audit was conducted to assess timeliness of evaluations. A mixed-method analysis was utilized, with nonparametric statistical testing and a constant-comparative qualitative method.ResultsThere was greater familiarity with the six core competencies than with Milestones or the NAS (p<0.05). A majority of faculty and residents felt that end-of-rotation evaluations were not adequate for surgical skills feedback. Fifty-eight per cent of residents reported that end-of-rotation evaluations were rarely or never filled out in a timely fashion. An internal audit demonstrated that more than 30% of evaluations were completed over a month after rotation end. Qualitative analysis included themes of resident desire for more face-to-face feedback on technical skills after operative cases, and several barriers to more frequent feedback.DiscussionThe NAS and outcome-based education have arrived. Residents and faculty need to be educated on this changing paradigm. This transition period is also a window of opportunity to address methods of evaluation and feedback. In our orthopaedic residency, trainees were significantly less satisfied than faculty with the amount of technical and surgical skills feedback being provided to trainees. The quantitative and qualitative analyses converge on one theme: a desire for frequent, explicit, timely feedback after operative cases. To overcome the time-limited clinical environment, feedback tools need to be easily integrated and efficient. Creative solutions may be needed to truly achieve outcome-based graduate medical education.