Documentation of Program Evaluation and Written Improvement Plan

The challenge of teaching ophthalmology: a residency review committee perspective

The Next Phase of Pediatric Residency Education: The Partnership of the Milestones Project

Residency training programs vary across characteristics such as their didactic and clinical experiences, attributes of the incoming residents, faculty characteristics, research conducted, community service performed by the program, and eventual practice choices of graduates. Which of these characteristics are most salient for evaluating the quality of a program? Elliott1 lists characteristics of graduates, cost—effectiveness, fair and ethical treatment of trainees, and meeting societal needs as important quality indicators. Iverson2 takes a dimensional approach. His dimensions, with metrics, are: intake [U.S. Medical Licensing Examination (USMLE) scores of matched applicants]; customer satisfaction [percentage of available positions filled by match, and percentage filled by U.S. medical school graduates (USMGs)]; residency review committee (RRC) quintile scores; and outcome (specialty board pass rates). The Accreditation Council for Graduate Medical Education (ACGME) recently switched its focus from process variables to outcomes, and is encouraging RRCs to evaluate a program on how well it provides for six core competencies: patient care, clinical science; interpersonal skills and communication, professionalism, practice-based learning and improvement, and systems-based practice.3 In 1997, a task force of the American Association of Directors of Psychiatry Residency Training (AADPRT) developed a survey to define the variables important to determining a program's quality from the psychiatry resident's perspective. The 41-item questionnaire was based on feedback from focus groups of psychiatry residents and program directors and a review of the literature. A total of 180 psychiatry residents from 16 programs completed the survey. Quality of supervision and teaching conferences, respect of faculty for residents, responsiveness of the program to feedback from residents, and morale in the department were the items most important to residents' satisfaction. A detailed description of the construction of the survey and its results was published by Elliott.4 In 1998, the AADPRT's survey was repeated with psychiatry residency directors and heads of major rotations to see whether their values agreed with those of the residents.5 This paper describes the use of multidimensional scaling (MDS) of the survey's results to establish whether there were distinct groupings of program directors with different opinions about the determinants of quality in psychiatry residency programs. These groupings might represent types of psychiatry programs (or market niches) as reflected in the values and priorities of their faculty and directors. Method Multidimensional scaling is an analytic technique frequently used in marketing research to identify the psychological dimensions underlying customers' preferences with respect to multiple variables or features of a product.6 In MDS the difference between clusters or groups of variables is predicted by the distance between the variables in psychological "space," with the dimensionality of the space equal to the number of relevant dimensions underlying the data. These dimensions can be thought of as analogous to the latent constructs derived in factor analysis. The scaling algorithm derives the dimensions and plots the coordinates of the variables in the resulting multidimensional space. MDS is an inherently interpretive procedure—it locates variables on dimensions but requires the investigator to determine whether the dimensions can be intelligibly labeled. Individual Differences Scaling (INDSCAL) is an MDS algorithm that models both the overall dimensions that underlie the perceptions of the group of respondents and individual weights on those dimensions for each respondent, allowing individuals to vary in the importance they attach to each of the dimensions. For example, for one individual, dimension A (the educational resources available, for example) may be highly salient, while dimension B (the administration of the program, for example) is relatively unimportant. For another individual, these priorities may be reversed. By examining the distribution of subject weights one can identify clusters of subjects who share similar values regarding the relative importances of the various dimensions. The questionnaire was sent in late 1998 to all psychiatry residency directors listed in the American Medical Association's 1998–1999 Directory of Accredited Graduate Medical Education (GME) Programs. The faculty members who served as the heads of the inpatient and outpatient psychiatry rotations of each program were also surveyed. These are the two major rotations of psychiatry training programs, and the opinions of the heads of these rotations (henceforth referred to as service chiefs) would most likely represent the dominant values of the program. The survey asked directors and service chiefs to rate how important the 41 items of the questionnaire were in determining the quality of a residency program. The anchors were 1 = least important, 4 = average importance, and 7 = most important. Multidimensional scaling using INDSCAL was done on the survey responses. Solutions in two to six dimensions were generated. Results Of the 186 active programs listed in the GME directory, 117 programs (63%) responded to the survey. There were 234 individual responses from the 117 programs. Of these, 142 (61%) were from program directors and 92 (39%) were from service chiefs who were not identified as directors. For some programs the head of inpatient or outpatient services also served as an associate program director, confirming our supposition that these faculty members represent the administrative backbone of the program. The Pearson correlation between the responses of the residency directors and those of the service chiefs was 0.98 (p < 0.01). We therefore pooled data from both chiefs' and residency directors' responses for the following analysis. The two-dimensional INDSCAL solution was degenerate and was discarded. The solutions in three to six dimensions were inspected for goodness of fit and interpretability. The three-dimensional configuration provided the most interpretable dimensions, and accounted for 46.4% of the variance in the data; higher-dimensional solutions accounted for only slightly more variance. Based on examination of the locations of the items, particularly those that had particularly high or low coordinates in each dimension, the dimensions seemed to correspond to three constructs: "curriculum," "quality of the institution," and "supportiveness of the administration of the program." The three dimensions, and the highest-loading items on each, are given in Table 1.TABLE 1: Three Dimensions of Quality of Psychiatry Residency Programs, Based on Multidimensional Scaling of Responses by Residency Directors and Service Chiefs to a 1998 Questionnaire*Subject weights measure the importance or salience of each dimension to each respondent; they range from 0 (completely ignored) to 1 (overwhelmingly important), and need not sum to 1. In addition, each respondent is assigned a "weirdness' value, which measures the similarity of his or her responses to those of the typical respondent, based on the relative importance of each dimension and the goodness of fit for that respondent. There was a great deal of variation in individual preferences, but no distinct clusters were evident. Notably, although weights for the supportiveness of the administration of the program dimension fell between 0.25 and 0.40 for nearly all respondents, the importance attributed to the dimensions of curriculum and quality of the institution varied extensively across individuals. Figure 1 plots the weights of curriculum and quality of the institution against one another for each respondent. The unshaded polygon encloses data for more typical respondents with less than the median weirdness, represented as circles; the two shaded polygons identify two groups of less typical respondents with more than the median weirdness, represented as crosses. Two respondents in the upper left had extreme (outlier) weirdness values.Figure 1: Subject weights for the "curriculum" and "quality of the institution" dimensions of the three-dimensional INDSCAL solution. The unshaded polygon encloses subjects with less than the median weirdness (based on weights in all three dimensions), represented as circles; the two shaded polygons identify two groups of subjects with greater than median weirdness, represented as crosses. Two subjects in the upper left had extreme (outlier) weirdness values."Weirdness" measures how similar each respondent is to the typical respondent, based on the relative importance of each dimension and the goodness of fit for that respondent.While the most typical respondents gave curriculum weights between 0.3 and 0.6, and quality of the institution weights between 0.2 and 0.4, two groups of respondents displayed different weight patterns. One group (lower right) gave curriculum substantially higher weights than typical (ranging from 0.5 to 0.75); the other group (upper left) gave quality of the institution substantially higher weights than typical (ranging from 0.25 to 0.65). On the average, most respondents' data displayed a continuum of weight patterns in which curriculum was considered to be more important than either quality of the institution or supportiveness of the administration of the program; the respondents with the lowest weirdness scores weighted these dimensions in the proportions 1.3:1:1, respectively. Discussion The three dimensions that emerged from the MDS are consistent with the many suggested quality indicators reviewed above.1,2,3 How might we conceptualize this triad? The dimensions of curriculum and supportiveness reflect two different aspects of the process of residency training. The curriculum dimension describes the content of the educational program; the supportiveness dimension reflects the culture or ambiance within which the training occurs. Residency directors and their faculty seem to differentiate between these two aspects of the program, and value both as indicators of the quality of the program. The dimension of institutional quality includes items reflecting the reputation and resources of the institution as well as items generally considered to be outcomes (i.e., board scores of graduates and graduates' job satisfaction). In this instance, these "outcome" items probably serve as proxies for (and a reflection of) the reputation of the institution and the quality of the residents it attracts, rather than as true outcome measures. This dimension could represent a general context factor, reflecting the quality of the facilities, the faculty, and of the residents themselves. This general factor could itself modify the effects of the process variables either up or down, thereby affecting the expected outcomes. Thus, this dimension may reflect the expectation of program directors and chiefs that equivalent processes (curriculum and supportiveness) could lead to better outcomes if they are provided in the context of a higher-quality institution. Donabedian lists input, process, and product as the dimensions of quality in healthcare.7 Interestingly, product (outcomes) did not emerge as a dimension of the quality of a program. Perhaps residency directors and service chiefs focus on process variables as indicators of quality since it is in the process of residency training that they deal. The neglect of outcome measures may also reflect a philosophy that, while the program is responsible for teaching, it is the residents' responsibility to learn. Outcome measures are highly confounded by the abilities and characteristics of the individual residents and, thus, may not be considered an accurate or reliable measure of the quality of the program per se. The ACGME and others who have begun the move towards outcome evaluation may wish to take this as a word of caution. Outcomes should not be considered in a vacuum. For at least some key stakeholders—the residents and faculty of the program—the context, content, and culture of the program are significant as well. No truly distinct clusters or groups of respondents emerged from the multidimensional scaling of the data. While there may be programs with different missions—programs oriented towards research or community psychiatry, for example—these missions do not seem to result in drastically different definitions of quality. This suggests that there is a core concept of quality that holds across contexts and across missions—consistent with the RRC's model of minimum standards. On the other hand, there seems to be a continuum of individual variation, rather than variation based on group membership. The individual respondents differed widely in the dimensions they considered most important. Since the individuals in this case are the faculty leaders and directors of the programs, these priorities are most likely reflected in the programs as a whole. The individual variation can be usefully segregated into three "market niches," corresponding to the three polygons in Figure 1. Thus we could describe three types of programs: (1) programs in which the quality of the sponsoring institution (context) is paramount, (2) programs in which the quality of the curriculum (content) is paramount, and (3) programs with a more typical weighting of the three dimensions. While there was less variability in the importance attached to the supportiveness (culture) of the program, this should not be construed as lack of salience. Rather, all programs should be alert to the importance of this dimension. Residents, too, vary in the levels of importance they attribute to the various features of a training program.4 The context, content, and culture of a program may provide a good conceptual model of the dimensions along which the market varies. Programs may find it useful to identify and market themselves on the basis of these dimensions. This study focused on only one of the many stakeholder groups of residency programs. The needs and expectations of other stake-holders, such as program funders and employers of a program's graduates, remain to be defined. This study also focused only on psychiatry programs, but the dimensions of context, content, and culture would seem to be potentially applicable to other specialties as well. Repeating the study with other specialties will tell us whether indeed this triad is relevant to the quality of programs across specialties. The method of multidimensional scaling is a novel one for determining quality measures in graduate medical education. Repeated use of this technique, across stakeholders and across specialties, can help elucidate the factors most important to the evaluation of residency programs.

This study sought to define expert opinion on the ideal length of training (LoT) for Accreditation Council of Graduate Medical Education (ACGME)-accredited emergency medicine (EM) residency programs. A cross-sectional Web-based survey was sent to program directors (PDs) at all ACGME-accredited EM residency programs during a study period of August to October 2014. The primary outcome of ideal LoT was determined in two ways: 1) subjects provided the ideal total LoT in months and 2) then separately selected the type and number of rotations for an ideal EM residency curriculum by month, the sum of which provided an alternative measurement of their ideal LoT. We did not include vacation time. Descriptive statistics and an analysis of variance are reported. Response rate was 68.0% (108/159) with 72% of respondents (78/108) directing programs in the PGY 1-3 (36-month) format and 28% directing PGY 1-4 (48-month) programs. More than half of subjects (51.9%) have direct personal experience with both formats. When asked about ideal total LoT, PDs averaged 41.5 months (n = 107; SD = 5.5 months, range = 36-60 months). When asked to provide durations of individual clinical experiences for their ideal EM program, the sum total (n = 104) averaged 45.0 months. Results from a factorial analysis of variance revealed statistically significant effects of PDs' past training experiences: participants who trained in a 36-month program had statistically significantly lower LoT (mean = 39.2 months) than participants who trained in a 48-month program (mean = 44.5 months). There was also a statistically significant effect of current program format on ideal LoT: participants who directed a 36-month program had statistically significantly lower LoT (mean = 39.8 months) than participants who directed a 48-month program (mean = 45.8 months). PD opinion on ideal LoT averages between 36 and 48 months, but is longer when the sum of desired clinical rotations is considered. While half of the respondents reported direct experience with both PGY 1-3 and PGY 1-4 training programs, opinions on ideal LoT through both methods corresponded strongly with the length of the program the PDs trained in and the format of the program they currently direct. PD opinions may be too biased by their own experiences to provide objective input on the ideal LoT for EM residency programs.

Certification by the American Board of Pediatrics (ABP) is a multistep process. The first step consists of the successful completion of a period of graduate education in pediatrics that is defined by the ABP and accomplished in a program accredited by the Residency Review Committee and the Accreditation Council for Graduate Medical Education, which are independent agencies ensuring separation of the evaluation of the training program (accreditation) from the evaluation of the individual (certification). The second step is recommendation of the applicant by the program director. The third step is a written examination that allows candidates to demonstrate their knowledge and, to some extent, their problem-solving skills. Passing the written examination is a requirement for the final step to certification, an oral examination. In this article, we describe the second step in the ABP's certification process, ie, evaluation and recommendation by the program director and how this relates to performance

In the Next Accreditation System, resident performance and progression assessed through educational Milestones will be used as 1 of a number of quality indicators to evaluate the educational effectiveness of residency programs. The Anesthesiology Milestones will be used to measure the success of individual anesthesiology residents in achieving expected goals of training, and the effectiveness of the residency program will in part be evaluated based on aggregated Milestone performance data.The members of the Anesthesiology Milestone Working Group were selected by the Residency Review Committee (RRC) for Anesthesiology, with input from the American Board of Anesthesiology (ABA), the Society for Education in Anesthesia, and program directors. The working group included 9 physician anesthesiologists, the executive director of the Anesthesiology RRC, and an Accreditation Council for Graduate Medical Education (ACGME) consultant. Four of the physician members were representatives from the Anesthesiology RRC (1 also served as the designated institutional official of a university-based medical center), 2 were former directors of the ABA, 2 were anesthesiology program directors, and 1 was an anesthesiology resident in the final year of training. An anesthesiologist who did not participate in the development of the Anesthesiology Milestones was appointed to the multidisciplinary expert panel that developed cross-specialty Milestones for interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice suggested for use by all specialties. Working group members were provided access to Milestone documents developed by several specialties that were a part of the initial stage in Milestone development. These models were used as templates to approach the task of developing anesthesiology-specific Milestones.The Anesthesiology Milestone Working Group held its first face-to-face meeting in February 2012, at which the ACGME consultant, who had experience working with prior Milestone working groups, provided an overview of the approach ACGME envisioned for the development of specialty-specific Milestones and shared lessons learned from Milestone development in the Phase 1 specialties. Based on the ACGME's recommendation that each Milestone should contain 5 levels of assessment (corresponding to the range from beginner to expert), initial discussions focused on determining if Milestone levels could be correlated with levels of residency training (clinical base year, clinical anesthesia years 1–3, and graduation). After recognizing that in their final form the levels would be independent of strictly time-based markers, the group made the decision that each Milestone level would represent expectations at the end of a point on the educational continuum. As a part of that continuum Level 1 Milestones were set as the expectations at the end of clinical base training. Level 2 Milestones were set to represent training expectations at the end of foundational (nonsubspecialty) clinical anesthesia training. Level 3 Milestones were set to represent the expectations for a resident at the midpoint of training after having been exposed to all subspecialty areas within anesthesiology. Level 4 Milestones were set to represent the expectations for a physician ready to practice anesthesiology independently without supervision. Level 5 Milestones were set to represent advanced characteristics of independently practicing physicians and were considered aspirational goals (a few of which might be achieved by a subgroup of exceptional residents during their training).After completion of the first draft of the Anesthesiology Milestones, the committee solicited feedback from members of the Association of Anesthesiology Core Program Directors (AACPD) and the Society for Education in Anesthesiology during their 2012 annual meetings. At the AACPD meeting feedback was provided formally through an audience response system that allowed program directors to respond to questions of relevance, importance, and measurability of each Milestone. Following these meetings, the working group edited the Milestones based upon the feedback received. This revised draft was then distributed to all program directors who were asked to provide constructive feedback through an electronic survey tool allowing confidential, individualized feedback from the program directors to the working group. The feedback was reviewed by the committee and used to make additional modifications to the Anesthesiology Milestones.The working group developed Milestones for each of the 6 ACGME competencies over the course of multiple face-to-face meetings and teleconferences. The patient care and medical knowledge Milestones, which are unique to each specialty, were developed first, and generic Milestones developed by a multispecialty panel were used as a starting point for the development of Milestones for the competencies of interpersonal and communications skills, problem-based learning and improvement, professionalism, and systems-based practice.The working group recognized the potential increase in workload that Milestone evaluation will place on program directors and Clinical Competency Committees (CCCs), and refined the number of Milestones to ensure appropriate assessment of competencies at each level of training, while not making the Milestones so granular that the evaluation process would be too burdensome. In the final version, the number of Milestones was reduced to 25. The working group believed this number provided an appropriate balance between comprehensiveness and practicality. The distribution of the Anesthesiology Milestones across the 6 competencies is shown in the table in this Introduction.While the working group sought input from the anesthesiology educational community to ensure that the Milestones were appropriate for assessing anesthesiology resident progression, the working group also recognized that the value of the proposed Milestones would benefit from an initial assessment using a pilot study in selected programs. A representative group of 20 anesthesiology programs has been selected to participate in the pilot study. The pilot will use the CCC in each program to assess the residents using the current Milestones. An evaluation tool has been developed for the program directors to provide the working group with feedback on each Milestone. The working group anticipates that the current version of the Anesthesiology Milestones will require modification over time as the specialty gains experience with this evaluation process. Once the final version of the Milestones is incorporated into training program evaluation, the RRC and other stakeholders will need to monitor and modify the Milestones based on experience so that they provide meaningful assessment of the residents, allowing anesthesiology training programs to foster excellence and optimize the educational experience for each trainee.The goal of the working group's effort was to produce Milestones that will provide program directors and CCC members with a progressive resident assessment tool that more readily identifies areas of strength and areas in need of improvement for each trainee. As Milestone data is accumulated in a department, the performance of an individual resident can be readily compared to peers and the course of progress of individual residents and residency levels in all 6 competencies domains can be evaluated. In addition, the program will have information about resident performance across the entire residency complement that will allow it to identify areas in which to focus educational efforts.The Next Accreditation System requires that accredited programs appoint CCCs to monitor residents' progress. The CCC is not new to anesthesiology training programs; the ABA has required submission of CCC reports for more than 25 years. The working group recognized that the Milestones represent a different and important approach to assessment of resident performance and progression. Although CCCs have been instrumental in assessing residents in anesthesiology programs, incorporation of Milestones into the assessment process will require modifications of the assessment process, faculty development to understand the Milestones and their implications and changes in the assessment, and communication with residents regarding performance.Based on the addition of Milestones in the assessment process, anesthesiology programs may have to modify the composition of the CCCs to ensure representation of all subspecialties and faculty involved in all levels of training. The ideal size of the CCC will likely vary by program based on the number of residents and faculty in the program. Regardless of the size of the CCC, faculty development will be the cornerstone in the transition to Milestones as a method for assessing clinical performance and progression through training. The working group recognized that as CCCs gain experience with incorporating Milestone assessments into their decisions, modifications of assessment strategies may be required. In addition, some Milestones may be difficult to assess in the traditional clinical environment because of the changing clinical environment and case distribution. Not all residents will be presented with the same clinical case experiences; therefore, a resident may not have an opportunity within the clinical environment to demonstrate some of the characteristics encompassed within the Milestones. The use of simulation, standardized patients, and other nonclinical assessments will likely grow in importance as methods for evaluating resident performance in some areas. The Milestone process will be helpful in assessing resident progression during training and the need for remediation when a resident is not progressing through a Milestone. At the same time, the Milestone process may be helpful in evaluating the effectiveness of and role for these alternative training methods in identifying areas of weakness.The adoption of Milestones in resident assessment will advance the journey toward competency-based graduate medical education that began with the Outcomes Project. Yet with this advance comes additional challenges. Once well-defined Milestones are fully integrated into residency education, it will be necessary to reconcile competency-based end points of training with the variable rate at which Level 4 Milestones will be reached by individual residents. Resolving this dilemma will provide opportunities for innovation in future training paradigms.

Traditional means of testing rheumatology fellows do not adequately assess some skills that are required to practice medicine well, such as humanistic qualities, communication skills, or professionalism. Institution of the New York City Rheumatology Objective Structured Clinical Examination (ROSCE) and our sequential 5 years of experience have provided us with a unique opportunity to assess its usefulness and objectivity as a rheumatology assessment tool. Prior to taking the examination, all of the fellows were rated by their program directors. Fellows from the participating institutions then underwent a multistation patient-interactive examination observed and rated by patient actors and faculty raters. Assessments were recorded by all of the participants using separate but overlapping sets of instruments testing the Accreditation Council of Graduate Medical Education (ACGME) core competencies of patient care, interpersonal and communication skills, professionalism, and overall medical knowledge. Although the program directors tended to rate their fellows more highly than the ROSCE raters, typically there was agreement between the program directors and the ROSCE faculty in distinguishing between the highest- and lowest- performing fellows. The ROSCE faculty and patient actor assessments of individual trainees were notable for a high degree of concordance, both quantitatively and qualitatively. The ROSCE provides a unique opportunity to obtain a patient-centered assessment of fellows' ACGME-mandated competencies that traditional knowledge-based examinations, such as the rheumatology in-service examination, cannot measure. The ability of the ROSCE to provide a well-rounded and objective assessment suggests that it should be considered an important component of the rheumatology training director's toolbox.

On July 1, 2013, the first group of programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) transitioned into the Next Accreditation System (NAS), and 1 year later, the remaining accredited specialties, the transitional year, and the review of sponsoring institutions will move into the NAS.1 The NAS is an outcomes-based and improvement-focused approach to the accreditation of programs and sponsoring institutions. The intent is to both ensure a level of quality (through assessment of compliance with the accreditation standards) and to promote ongoing improvement in all programs and innovation in high-performing programs.1 The added focus on improvement will be achieved through (1) a Self Study that each program will complete before its 10-year site visit and includes aggregation and analysis of data on its longitudinal improvement effort, and (2) a Self Study site visit during which the program will describe its improvement process and the outcomes achieved. The Self Study site visit will occur approximately 1 year after the program begins its Self Study.The lengthening of site visit intervals in the NAS and the focus on outcomes and continuous improvement call for a new type of site visit, and this new model requires testing. To further inform the Self Study and to increase its relevance to programs, the ACGME envisioned a new focus on programs' aims, combined with an environmental assessment that focuses on strengths, areas for improvement, opportunities, and threats.A key aim in the NAS is to reduce the burden of accreditation, and a recognized source of burden is the preparation of site visit documentation. Thus, the Self Study site visit also calls for a new approach to reporting that provides insight into programs' longitudinal improvement efforts, without that documentation either adding undue burden or affecting the ability of the program to conduct a complete and honest accounting of its strengths and its areas in need of improvement.The Self Study approach is currently being refined, with the first Self Studies for programs in Phase I of the NAS scheduled to begin in 2014, and the first Self Study site visits planned for 2015. In this brief article, we offer early information from the ongoing, conceptual testing of the Self Study site visit.By 2012, the aims and content of the 10-year Self Study and Self Study site visit (box 1) had evolved sufficiently to allow the ACGME to test the new elements of the proposed approach. This included the enhanced focus on program improvement, using the Annual Program Evaluation, and the new focus on program aims and an assessment of programs' performance and opportunities and threats in their local environment.In appreciation of the significant development the Self Study process and the new type of site visit would require, ACGME leadership thought it would be beneficial to test the emerging concepts and approaches in a sample of Self Study test visits, beginning in late 2012. It was clear that a “real” Self Study site visit would have required the programs to have completed a self-assessment and that the test visits would test only selected elements of the planned approach.To date, site visits at 3 institutions involving internal medicine and plastic and vascular surgery core programs and subspecialty programs in those areas have been completed. Test visits involved volunteer institutions, and the findings were not shared with Residency Review Committees. All test site visit programs were visited by a team, as is planned for the actual Self Study site visits. The individuals interviewed and the time commitment for participants on the day of the site visit were comparable to a “regular” site visit. The ACGME sought to ensure that preparation before a site visit would be minimal, and the site visit was limited to asking the residents or fellows in the test programs to complete a consensus list of program strengths and areas for improvement.The test visit entailed a debriefing with site visitors and program participants and produced reports for internal sharing and comprehensive field notes. In contrast to the more general compliance focus that had characterized earlier tests of Program Information Form-less site visits,2 these visits tested selected new elements planned for future Self Study visits. They included a focus onItems 2 through 4 above constituted an initial test of new dimensions the ACGME envisioned as part of a future NAS 10-year Self Study site visit. One of the key information items for this portion of the visit was the Annual Program Evaluation. To continue to preserve the ability of programs to conduct a frank assessment of areas for improvement, the test visits maintained the confidentiality of the Annual Program Evaluation, and the team asked the program director to discuss program improvement, articulating successes achieved and areas still being worked on, based on the several recent Annual Program Evaluations. The ACGME envisions that a similar approach will be used for actual Self Study visits.Although the sample is small, the test site visits to date have confirmed the feasibility and “assess-ability” of the added dimensions of a Self Study visit. Programs articulated “aims” without difficulty, although, other than efforts to address lower than desirable board performance, offering general information on preparing residents for academic careers, and in 1 instance, starting a primary care track, respondents were less clear about their ongoing activities to enhance performance relevant to these aims.Discussion during the assessment of strengths, areas for improvement, opportunities, and threats focused primarily on national threats, particularly changes in federal support for graduate medical education (GME). Local opportunities and threats were less readily identified in the conversations with program leadership. This suggests that these new site visit dimensions would benefit from ongoing assessment over time, and it is expected that these areas would be more developed in a “real” Self Study, which would have been preceded by a formal Self Study process.The early test site visits also highlighted several areas of emerging focus, some likely prompted by the increased emphasis on educational outcomes in the NAS. During the site visits, programs discussed new modes of evaluation on the Milestones; on initiatives to restructure their didactic programs, focusing on scheduling and delivery models to maximize access and participation; and on measuring the effect of the didactic offerings. Preparation for the board examination, as the most visible element of outcomes focus in the current system, constituted a major focus of the discussion of educational outcomes across programs at the programs sites visited to date. Another focal area likely resulted from new roles for faculty in the NAS, particularly professional development for assessing performance on the milestones and participation on Clinical Competency Committees. In addition to faculty development content, discussions addressed approaches to encourage participation, including scheduling strategies and financial and other incentives, such as academic promotion and recognition.Finally, the visits highlighted some beneficial approaches for tracking data on longitudinal improvement, using simple, common institutional formats that allowed programs to record and report their improvements and areas still being worked on. This longitudinal tracking of improvement will gain in importance in the NAS, as much of the focus of the Self Study visit will be on programs' effectiveness in making ongoing improvements.The early tests contributed significantly to the development and refinements of the program Self Study approach and the associated site visits envisioned for the NAS. This work will continue with a focus on (1) offering guidance to programs about effective and efficient approaches for longitudinal improvement, including sessions at the ACGME Annual Educational Conference; (2) developing a report that aggregates NAS performance data the ACGME would provide to programs at the “kick-off” of their Self Study effort, approximately 1 year before the scheduled site visit; and (3) formulating a succinct document for programs to complete in preparation for a Self Study site visit.During the early test visits, it became clear that a truly meaningful Self Study visit would require the prior completion of a Self Study process. The concept is widely used in educational accreditation, yet it remains one of the more debated elements. The ideal approach for an NAS Self Study would produce a self-assessment that aids in the acquisition of knowledge about the program and identifies areas of real, tangible improvements.3 At the same time, organizational and programmatic Self Studies, although valuable from an improvement perspective, carry significant time, financial, and opportunity costs.4 It is clear that, given the smaller size of residency programs or even a core programs and its associated subspecialty programs compared with universities or medical schools and the resource constraints in the GME environment, the Self Study approach will need to be adapted and streamlined for use in the NAS. It also became apparent that the process would need to address scheduling challenges for programs, particularly when core programs and their subspecialty programs are seen as a group, as is planned in the NAS. The rationale and anticipated benefits for having a collective Self Study and Self Study site visit for core programs and their associated subspecialty programs are shown in box 2.The exploration of elements of the Self Study and the associated visit confirmed the feasibility of the approach and acceptance by program and institutional participants, although the sample was limited and the use of the information for internal improvement may have produced a more receptive and less critical audience. Future challenges and areas to further develop and refine the site visit approach in the NAS will need to deal with efforts to assess programs' ongoing focus on improvement while ensuring the confidentiality of the data underlying that process.Additional site visits are planned, with a particular focus on tapping into programs' ongoing improvement efforts, and how those efforts could be documented without impinging on their ability to conduct frank and forthright assessments of strengths and areas for improvement. Future work will also explore formats that offer added value to programs, such as a Self Study or site visit that yields valuable insight for programs in key areas, particularly recognition of opportunities and threats within their environment.

Coaching for professional development contributes to improved performance, personal satisfaction, and well-being. Although coaching programs have been widely introduced at the undergraduate medical education level and in certain residency programs, coaching programs targeting the unique experiences of fellows are less well described. Given high rates of burnout, pulmonary and critical care medicine (PCCM) fellows may benefit from professional development coaching. To design, implement, and evaluate an individual coaching program for PCCM fellows. Alumni surveys and trainee and faculty focus groups identified key needs of a coaching program. Coaches received salary support and were trained in competencies including positive psychology, learner engagement, and communication. Starting in 2019, three coaches were assigned seven fellows each, spanning the three Accreditation Council of Graduate Medical Education (ACGME) years. The program was evaluated through ACGME surveys, Likert scale-based end-of-year surveys, and qualitative analysis of fellows' written feedback. Coaching was associated with improved overall training program satisfaction (ACGME survey overall satisfaction (survey range 1-5 with 5 highest) was 4.2, 3.9, 4.3 in the three years pre-intervention, and 4.6, 4.6, 4.6 in the three years post-intervention). Compliant responses in the ACGME survey domain of Evaluation increased above our program benchmark of 90% from a baseline of 75%. Fellow well-being demonstrated slight improvements through the evaluation period (2018-2022). Fellows rated the coaching program highly, especially in support and well-being, feedback, and goal-setting. Only 66% reported a direct impact on choosing a specific career path. Implementation of this fellowship coaching program was associated with improvements in fellows' satisfaction with the training experience, evaluation and feedback, career development, and well-being, despite the added challenges of the COVID-19 pandemic. Future directions include standardizing coaching frameworks, exploring long-term impacts on career trajectories, and evaluating coaches' satisfaction with the program.

Clinical experiences, quantified by case logs, are an integral part of pediatric anesthesiology fellowship programs. Accreditation of pediatric anesthesiology fellowships by the Accreditation Council of Graduate Medical Education (ACGME) and establishment of case log reporting occurred in 1997 and 2009, respectively. The specialty has evolved since then, but the case log system remains largely unchanged. The Pediatric Anesthesiology Program Directors Association (PAPDA) embarked on the development of an evidence-based case log proposal through the efforts of a case log task force (CLTF). This proposal was part of a larger consensus-building process of the Society for Pediatric Anesthesia (SPA) Task Force for Pediatric Anesthesiology Graduate Medical Education. The primary aim of case log revision was to propose an evidence-based, consensus-driven update to the pediatric anesthesiology case log system. This study was executed in 2 phases. The CLTF, composed of 10 program directors representing diverse pediatric anesthesiology fellowship programs across the country, utilized evidence-based literature to develop proposed new categories. After an approval vote by PAPDA membership, this proposal was included in the nationally representative, stakeholder-based Delphi process executed by the SPA Task Force on Graduate Medical Education. Thirty-seven participants engaged in this Delphi process, during which iterative rounds of surveys were used to select elements of the old and newly proposed case logs to create a final revision of categories and minimums for updated case logs. The Delphi methodology was used, with a two-thirds agreement as the threshold for inclusion. Participation in the Delphi process was robust, and consensus was almost completely achieved by round 2 of 3 survey rounds. Participants suggested that total case minimums should increase from 240 to 300 (300-370). Participants agreed (75.86%) that the current case logs targeted the right types of cases, but requirements were too low (82.75%). They also agreed (85.19%) that the case log system and minimums deserved an update, and that this should be used as part of a competency-based assessment in pediatric anesthesia fellowships (96%). Participants supported new categories and provided recommended minimum numbers. The pediatric anesthesiology case log system continues to have a place in the assessment of fellowship programs, but it requires an update. This Delphi process established broad support for new categories and benchmarked minimums to ensure the robustness of fellowship programs and to better prepare the pediatric anesthesiology workforce of the future for independent clinical practice.

An internet-based residency assessment application that fulfills the outcome project’s requirements

Fellowship Training in Adult Cardiothoracic Anesthesiology: Navigating the New Educational Landscape as a Result of the Coronavirus Crisis

The Accreditation Council for Graduate Medical Education's Next Accreditation System requires continuous program improvement as part of program evaluation for residency training institutions and programs. To improve the institutional- and program-level evaluation processes, to operationalize a culture of continuous quality improvement (CQI), and to increase the quality and achievement of action items, the Wayne State University Office of Graduate Medical Education (WSU GME) incorporated CQI elements into its program evaluation process. Across 4 academic years, WSU GME phased the following 4 CQI elements into the evaluation process at the program and institutional levels, including the annual program evaluation (APE) and the annual institutional review: (1) An APE template; (2) SMART (specific, measurable, accountable, realistic, timely) format for program and institutional goals; (3) Dashboard program and institutional metrics; and (4) Plan-do-study-act cycles for each action item. Action item goals improved in adherence to the SMART format. In 2014, 38% (18 of 48) omitted at least 1 field, compared with 0% omitting any fields in 2018. More complete action items took less time to resolve: 1.7 years compared with 2.4 years (t (43.3) = 2.87, P = .003). The implementation of CQI in the APE was well received by program leadership. After leveraging CQI methods, both descriptions of institutional- and program-level goals and the time required for their achievement improved, with overall program director and program coordinator satisfaction.

The evaluation of Competence by Design (CBD) residency programs is crucial for enhancing program effectiveness. However, literature on evaluating CBD programs is limited. We conducted a 2-phase mixed-methods study to (a) assess the extent of program evaluation activities in CBD residency programs in Canada, (b) explore reasons for engaging or not engaging in these activities, (c) examine how CBD programs are conducting program evaluations, and (d) identify ways to build capacity for program evaluation. Phase 1 involved surveying 149 program directors from specialty/subspecialty programs that transitioned to CBD between 2017 and 2020. We calculated descriptive statistics for 22 closed-ended survey items. Phase 2 comprised interviews with a subset of program directors from Phase 1. Data analysis followed a 3-step iterative process: data condensation, data display, and drawing and verifying conclusions. In Phase 1, we received 149 responses, with a 33.5% response rate. Of these, 127 (85.2%) indicated their programs engage in evaluation, while 22 (14.8%) do not. Among the 127 programs that engage in evaluation, 29 (22.8%) frequently or always develop evaluation questions, and 23 (18.1%) design evaluation proposals/plans. Reasons for engaging in evaluation included decision-making and stimulating changes in educational practices. Conversely, reasons for not engaging included lack of knowledge, personnel, and funding. In Phase 2, 15 program directors were interviewed. They reported that CBD programs face challenges such as limited resources and buy-in, rely on ad hoc evaluation methods, and use a team-based evaluation format. To enhance evaluation capacities, interviewees suggested (a) developing expertise in program evaluation, (b) acquiring evaluation resources, and (c) advocating for clear evaluation expectations. Most CBD residency programs are engaged in program evaluations, but the quality is often questionable. To fully realize the potential of program evaluation, CBD programs need additional resources and support to improve evaluation practices and outcomes.

At the conclusion of my presidential address describing the events that led to a subspecialty certificate in Advanced Surgical Oncology at the 63rd Annual Cancer Symposium of the Society of Surgical Oncology (SSO) in March 2010, I stated: ‘‘It is fair to say that the goal is in sight, yet it is prudent to remind ourselves that we have not crossed the finish line yet’’. 3 years later, I have realized how prophetic that statement was. The events that followed the submission of a formal request by the American Board of Surgery (ABS) to the American Board of Medical Specialties (ABMS) to establish a certificate for Advanced Surgical Oncology and the subsequent steps that led to its approval by the ABMS have been detailed in a previous editorial. This was an historical event for the discipline of surgical oncology, which was recognized as deserving of a formal training and a unique certificate. This was also a significant moment for the ABS in that this certificate is the first new certificate offered by the ABS in more than 20 years. The process then continued with the Accreditation Council on Graduate Medical Education (ACGME). The ACGME asked the Residency Review Committee in Surgery (RRC-S) to specify the training requirements for surgical oncology programs. A subcommittee of the RRCS was established with Drs. Peter Fabri, Mark Wallack, and Thomas Whalen (chair). The SSO was asked to name a representative, and Dr. Russell Berman joined the subcommittee. Once defined, the proposed program requirements were then sent to the GME constituency on February 6, 2012, for the statutory 45-day comment period. After the RRC-S collated and answered all comments received, it submitted the final document to the Requirement Development Committee of the ACGME Board of Directors for review. Finally, the full Board of Directors of the ACGME approved the program requirements at the June 10, 2012, meeting. By the first week in July, both the program requirements and the program information form (PIF) were available on the ACGME Web site. In recognition of the high quality of program supervision that the Training Committee of the SSO had provided to SSO-accredited programs, the RRC-S, which has oversight responsibility for the fellowships, agreed that current SSOapproved programs would not require a site visit as a part of the initial accreditation review. Non-SSO-approved programs would require a site visit to verify and confirm the information provided in the PIF. In accordance with ACGME policy, programs accredited in the 2012–2013 academic year could request a retroactive effective date of accreditation to July 1, 2012, making graduates of these ACGME-accredited fellowships eligible to participate in the ABS certification process upon graduation on June 30, 2013. In the meantime, changes also occurred at the ABS. The name of the Surgical Oncology Advisory Council (SOAC) was changed to Surgical Oncology Board. In addition, its composition was defined (Table 1). I was elected the first chair of the Surgical Oncology Board (2011–2012) and was then followed by Dr. Selwyn Vickers, who is currently serving a 2-year mandate. The newly established Surgical Oncology Board decided to award the certificate of Complex General Surgical Oncology to eligible candidates after successfully passing a qualifying examination (QE) and a certifying examination (CE), very much like the process leading to a certificate in surgery. The first order of business therefore became to establish a sizable multiplechoice question bank for the qualifying exam. In order to expeditiously attend to this task, the Surgical Oncology Board created a group of 12 consultants, two for Society of Surgical Oncology 2013