Natural Language Processing to Estimate Clinical Competency Committee Ratings

In the Next Accreditation System, resident performance and progression assessed through educational Milestones will be used as 1 of a number of quality indicators to evaluate the educational effectiveness of residency programs. The Anesthesiology Milestones will be used to measure the success of individual anesthesiology residents in achieving expected goals of training, and the effectiveness of the residency program will in part be evaluated based on aggregated Milestone performance data.The members of the Anesthesiology Milestone Working Group were selected by the Residency Review Committee (RRC) for Anesthesiology, with input from the American Board of Anesthesiology (ABA), the Society for Education in Anesthesia, and program directors. The working group included 9 physician anesthesiologists, the executive director of the Anesthesiology RRC, and an Accreditation Council for Graduate Medical Education (ACGME) consultant. Four of the physician members were representatives from the Anesthesiology RRC (1 also served as the designated institutional official of a university-based medical center), 2 were former directors of the ABA, 2 were anesthesiology program directors, and 1 was an anesthesiology resident in the final year of training. An anesthesiologist who did not participate in the development of the Anesthesiology Milestones was appointed to the multidisciplinary expert panel that developed cross-specialty Milestones for interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice suggested for use by all specialties. Working group members were provided access to Milestone documents developed by several specialties that were a part of the initial stage in Milestone development. These models were used as templates to approach the task of developing anesthesiology-specific Milestones.The Anesthesiology Milestone Working Group held its first face-to-face meeting in February 2012, at which the ACGME consultant, who had experience working with prior Milestone working groups, provided an overview of the approach ACGME envisioned for the development of specialty-specific Milestones and shared lessons learned from Milestone development in the Phase 1 specialties. Based on the ACGME's recommendation that each Milestone should contain 5 levels of assessment (corresponding to the range from beginner to expert), initial discussions focused on determining if Milestone levels could be correlated with levels of residency training (clinical base year, clinical anesthesia years 1–3, and graduation). After recognizing that in their final form the levels would be independent of strictly time-based markers, the group made the decision that each Milestone level would represent expectations at the end of a point on the educational continuum. As a part of that continuum Level 1 Milestones were set as the expectations at the end of clinical base training. Level 2 Milestones were set to represent training expectations at the end of foundational (nonsubspecialty) clinical anesthesia training. Level 3 Milestones were set to represent the expectations for a resident at the midpoint of training after having been exposed to all subspecialty areas within anesthesiology. Level 4 Milestones were set to represent the expectations for a physician ready to practice anesthesiology independently without supervision. Level 5 Milestones were set to represent advanced characteristics of independently practicing physicians and were considered aspirational goals (a few of which might be achieved by a subgroup of exceptional residents during their training).After completion of the first draft of the Anesthesiology Milestones, the committee solicited feedback from members of the Association of Anesthesiology Core Program Directors (AACPD) and the Society for Education in Anesthesiology during their 2012 annual meetings. At the AACPD meeting feedback was provided formally through an audience response system that allowed program directors to respond to questions of relevance, importance, and measurability of each Milestone. Following these meetings, the working group edited the Milestones based upon the feedback received. This revised draft was then distributed to all program directors who were asked to provide constructive feedback through an electronic survey tool allowing confidential, individualized feedback from the program directors to the working group. The feedback was reviewed by the committee and used to make additional modifications to the Anesthesiology Milestones.The working group developed Milestones for each of the 6 ACGME competencies over the course of multiple face-to-face meetings and teleconferences. The patient care and medical knowledge Milestones, which are unique to each specialty, were developed first, and generic Milestones developed by a multispecialty panel were used as a starting point for the development of Milestones for the competencies of interpersonal and communications skills, problem-based learning and improvement, professionalism, and systems-based practice.The working group recognized the potential increase in workload that Milestone evaluation will place on program directors and Clinical Competency Committees (CCCs), and refined the number of Milestones to ensure appropriate assessment of competencies at each level of training, while not making the Milestones so granular that the evaluation process would be too burdensome. In the final version, the number of Milestones was reduced to 25. The working group believed this number provided an appropriate balance between comprehensiveness and practicality. The distribution of the Anesthesiology Milestones across the 6 competencies is shown in the table in this Introduction.While the working group sought input from the anesthesiology educational community to ensure that the Milestones were appropriate for assessing anesthesiology resident progression, the working group also recognized that the value of the proposed Milestones would benefit from an initial assessment using a pilot study in selected programs. A representative group of 20 anesthesiology programs has been selected to participate in the pilot study. The pilot will use the CCC in each program to assess the residents using the current Milestones. An evaluation tool has been developed for the program directors to provide the working group with feedback on each Milestone. The working group anticipates that the current version of the Anesthesiology Milestones will require modification over time as the specialty gains experience with this evaluation process. Once the final version of the Milestones is incorporated into training program evaluation, the RRC and other stakeholders will need to monitor and modify the Milestones based on experience so that they provide meaningful assessment of the residents, allowing anesthesiology training programs to foster excellence and optimize the educational experience for each trainee.The goal of the working group's effort was to produce Milestones that will provide program directors and CCC members with a progressive resident assessment tool that more readily identifies areas of strength and areas in need of improvement for each trainee. As Milestone data is accumulated in a department, the performance of an individual resident can be readily compared to peers and the course of progress of individual residents and residency levels in all 6 competencies domains can be evaluated. In addition, the program will have information about resident performance across the entire residency complement that will allow it to identify areas in which to focus educational efforts.The Next Accreditation System requires that accredited programs appoint CCCs to monitor residents' progress. The CCC is not new to anesthesiology training programs; the ABA has required submission of CCC reports for more than 25 years. The working group recognized that the Milestones represent a different and important approach to assessment of resident performance and progression. Although CCCs have been instrumental in assessing residents in anesthesiology programs, incorporation of Milestones into the assessment process will require modifications of the assessment process, faculty development to understand the Milestones and their implications and changes in the assessment, and communication with residents regarding performance.Based on the addition of Milestones in the assessment process, anesthesiology programs may have to modify the composition of the CCCs to ensure representation of all subspecialties and faculty involved in all levels of training. The ideal size of the CCC will likely vary by program based on the number of residents and faculty in the program. Regardless of the size of the CCC, faculty development will be the cornerstone in the transition to Milestones as a method for assessing clinical performance and progression through training. The working group recognized that as CCCs gain experience with incorporating Milestone assessments into their decisions, modifications of assessment strategies may be required. In addition, some Milestones may be difficult to assess in the traditional clinical environment because of the changing clinical environment and case distribution. Not all residents will be presented with the same clinical case experiences; therefore, a resident may not have an opportunity within the clinical environment to demonstrate some of the characteristics encompassed within the Milestones. The use of simulation, standardized patients, and other nonclinical assessments will likely grow in importance as methods for evaluating resident performance in some areas. The Milestone process will be helpful in assessing resident progression during training and the need for remediation when a resident is not progressing through a Milestone. At the same time, the Milestone process may be helpful in evaluating the effectiveness of and role for these alternative training methods in identifying areas of weakness.The adoption of Milestones in resident assessment will advance the journey toward competency-based graduate medical education that began with the Outcomes Project. Yet with this advance comes additional challenges. Once well-defined Milestones are fully integrated into residency education, it will be necessary to reconcile competency-based end points of training with the variable rate at which Level 4 Milestones will be reached by individual residents. Resolving this dilemma will provide opportunities for innovation in future training paradigms.

The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project has shifted the focus for accrediting residency training programs from structure and process to documented outcomes. A central part of the ACGME's Next Accreditation System (NAS) is “articulating milestones of competency development in each discipline.”1 Resident achievement of educational Milestones will be an indicator of the educational effectiveness of residency programs. Most specialties have developed their Milestones through a collaborative effort involving representatives from the ACGME, the pertinent American Board of Medical Specialties (ABMS) organization, the pertinent professional specialty society, and the pertinent program directors' association. In contrast, the approach was quite different for Transitional Year (TY) residency programs, which comprise only 1 year of accredited postgraduate medical education, where there is no corresponding ABMS board certification entity or professional specialty society. In addition, graduates of TY training programs are not expected to demonstrate sufficient competency to enter the independent practice of medicine, as is the case with other specialty (categorical) residency programs.

The goal of postgraduate training is the development of competency necessary for independent practice. To achieve this goal, the Accreditation Council for Graduate Medical Education (ACGME) in 2013 implemented the Milestones assessment framework and mandated the development of clinical competency committees (CCCs), tasked with biannual review of evaluation and assessment data of trainees.1 The CCC, which consists of a chair and committee members, plays a critical role in assisting the program director in decision-making regarding resident or fellow advancement through the training period. Importantly, the committee makeup may span junior and senior faculty and include interprofessional representation, thus providing a spectrum of input.1To date, most of the literature regarding CCCs focuses on optimizing CCC function in terms of workflow and decision-making. Best practices for CCC function have been proposed, including use of varied assessment tools, structured meetings, and diverse members.2,3 Other work has outlined strategies for effective group decision-making necessary for CCC function.4–6 Ongoing faculty development has been proposed as an essential component of CCC function, yet little is known about which particular elements are necessary to optimize organization of meetings, decision-making, and to effectively utilize assessment processes.In order for the CCC to optimally operationalize the Milestones for individual learner development, members of the committee require intentional faculty development. This must span multiple domains important for the CCC process, including knowledge of assessment (and the use of the Milestones), understanding of group decision-making, awareness of how biases interplay with these factors, and the effect of their decisions on patient care (Box). While it remains unknown how often and who should develop such faculty development sessions, we would advocate for at least an annual review of these topics given faculty turnover and need to maintain a shared mental model of best practices. Faculty development remains essential for both new and longstanding members, as participants will likely have different levels of understanding of the assessment process and Milestones themselves.2We therefore aim to provide guidance for specific areas of faculty development for CCC members, including strategies for novel approaches to professional development for the committee. This includes: (1) the purpose and structure of the CCC itself; (2) the strategies, strengths, and weaknesses of the assessments employed by the program; and (3) the development of a shared mental model of the Milestones of the relevant specialty. Beyond this, the CCC chair will also need to understand (4) the best practices of an effective group discussion.7Faculty need to understand the purpose of the CCC meetings, which often employ 1 of 2 strategies: problem identification, where the committee focuses on identifying learners who may require additional remediation, or a developmental approach, where the committee discusses all learners to identify next steps for advancement.3 Members also need to understand the process of decision-making, including who will make the recommendations regarding remediation.In addition to understanding the intent of the Milestones, committee members must also be aware of the program's evaluation process (as related to each Milestone). Knowledge of the individual assessment methods and awareness of the data synthesis for each Milestone will improve a shared mental model and ideally minimize differences in individual interpretation of assessment data.6 This will also allow the committee to identify times when the assessment data is difficult to interpret requiring further group discussion.8Committee members should not only understand each of the relevant Milestones but also have a shared model of what a competent trainee looks like at their institution.9 It is evident that competencies are developed differently in different contexts and at different institutions.10 Therefore, a shared mental model is important for each CCC to minimize variation in how each member interprets the same assessment data.Effective committee groups understand the purpose, structure, and process of their work.4 However, sometimes groups encounter incomplete or otherwise difficult to interpret assessment data; therefore, group discussion is needed to draw a conclusion. The chair of the committee needs to lead the discussion and approach how decisions are made transparently. The chair should also be cognizant of the member composition of the group.4 A homogenous group is unlikely to present different views or opinions, which may be necessary in drawing fair conclusions.Additional faculty development topics for the CCC chair may be helpful in maximizing success of the committee. It is incumbent upon the chair to refine their own ability to synthesize feedback as well as interpret and gauge performance in specialty-specific Milestones. General leadership skills, such as how to effectively lead a committee or run a meeting, remain paramount.11,12 It is important for the CCC chair to adapt their mindset from the frame of educator to that of the “educator-leader.” The CCC chair plays a pivotal role in ensuring that the committee functions through fair and informed decision-making processes based on the data and facts, rather than emotional decision-making.Beyond this unique CCC training for the chair and members, all faculty educators must understand the intersection of biases on the assessment process. It is well-known that biases can manifest in a variety of ways through our traditional mechanisms of assessment. For example, in graduate medical education (GME), this has been highlighted in several settings showing differential Milestone achievement based on trainee gender,13–15 as well as gender differences in narrative evaluation.16 Similarly, standardized patient assessments, which are often included in CCC discussions, have highlighted differential findings by gender, race, and ethnicity.17 Knowledge of such biases and the potential ramifications on Milestones ratings, narrative evaluation, remediation referrals, and progression decisions is imperative for faculty development in the CCC process.Faculty development using bias reduction workshops or training, such as those employing the Implicit Association Test, could represent an avenue to initiate discussions,18 as these training programs have shown to be effective in mitigating biased decision-making in other settings.18–20 The potential quantitative and qualitative differences in assessment should be considered in recurring CCC reviews. Additionally, faculty participants should be versed in the potential impact of system-wide and personal biases on trainee assessments. Beyond an individual training level, CCC structure, including the diversity of CCC members, as well as process measures, including program assessment and trends, should be regularly reviewed for biased tendencies.2There are a multitude of topics important in faculty development for CCC chairs and members and implementation often requires novel solutions. One such approach was recently implemented within the MedStar Health GME Consortium in the Baltimore, MD and Washington, DC region, which includes more than 70 ACGME-accredited programs. The consortium conducted a half-day faculty development retreat specifically for CCC chairs. Topics addressed included a primer on evaluation of competence, a panel discussion of CCC best practices (for varied sizes of programs), small group discussions on CCC decision-making processes, and strategies for running effective meetings. The program also allowed for identification of distinct needs for various CCC stakeholders, as outlined in the Table. These sessions allowed for shared knowledge, highlighted knowledge gaps, and identification of distinct faculty training needs for success of CCCs. Such an approach allowed for sharing of best practices, strategies to overcome real or perceived challenges (such as time constraints to effective discussion in larger scale programs), and dissemination of knowledge throughout the sponsoring institution.Overall, the optimal functioning of a CCC requires dedicated faculty time, education, and investment in faculty development. CCCs form a critical component of GME assessment and play a pivotal role in ensuring readiness for independent practice. Given this critical role, we advocate for an appropriate fraction of protected time (or productivity) from clinical or other administrative duties for CCC chairs and similar key leaders to accomplish the important responsibilities of the role.We have highlighted some overarching themes of essential faculty development for CCC success, but recognize that the needs are varied based on role within the CCC, experience level, and size of the program, among other considerations. It is critical to acknowledge differing faculty development needs across levels of experience of faculty (junior versus senior members), as well as the unique training needs of interprofessional members. Additionally, future scholarship is needed to determine the optimal timing, content, and subsequently the impact of such faculty development on CCC processes. We believe that the continued development of expertise for both the CCC chair and its members remain critical to the success of the Milestones assessment system.

The Milestones Development team at the Accreditation Council for Graduate Medical Education (ACGME) has periodically surveyed the graduate medical education (GME) community and volunteers, during and after development of the specialty-specific Milestones 2.0, to check with members of each specialty on the implementation of the new Milestones. As we approached the conclusion of Milestones 2.0 development, it was decided to have broader survey dissemination for all specialties. Although still only 10 questions, 2 major changes were made to the quality assurance feedback questionnaire: (1) It solicited comments and impressions from the approximately 26 000-member email distribution list that receives the ACGME e-communications rather than only those who volunteer to be respondents; and (2) It had a more summative focus meant to be the start of the upcoming program evaluation stage of Milestones, rather than soliciting formative input for any further changes to Milestones 2.0. The survey was open for 6 weeks in the fall of 2022 and resulted in 215 responses, which, after clearing the data of collection errors and detectable duplications, produced 211 valid responses from 27 specialties across the GME community. At the time of distribution, email and survey tracking metrics were not available to produce a meaningful response rate. Respondents provided general descriptions of their specialties and specific roles, their program enrollment, and the membership of their Clinical Competency Committees (CCCs). They were then asked open-ended questions that allowed them to describe any challenges with Milestones 2.0 and to make suggestions for the program as a whole. The intent of this 10-item questionnaire was to collect data that will direct conversations on change and help guide this first step, as well as further program evaluation projects that explore the relationships between Milestones and the GME community. The results of the qualitative analysis of this questionnaire are based on those who chose to respond to the ACGME e-communication request for feedback. The subsequent findings should not be generalized to the GME community or be made to represent thoughts or reactions of any subpopulation therein.Of 211 valid responses, roughly half (n=105, 49.8%) identified as program directors (Table 1) from a dropdown list of options (Question [Q]3), and 46 (21.8%) indicated that they were their CCC chair (Q4). The most represented ACGME core specialties (Q1) were internal medicine (n=46, 21.8%), family medicine (n=35, 16.6%), pediatrics (n=24, 11.4%), and surgery (n=17, 8.1%). Additionally, 80 (37.9%) respondents self-identified a subspecialty (Q2) in the open text question following the specialty question.The responses to (Q5) “Indicate the number of residents or fellows you are currently assessing in your program using the Milestones” ranged from 2 to 170. This question was asked to obtain an approximate number of residents and fellows being assessed, and it was beyond the purview of this project to check a respondent's accuracy regarding this data point.When asked to describe the roles within their current CCC membership (Q6), 181 (85.8%) of the respondents listed their committee as having a program director, 162 (76.8%) indicated a program coordinator, 161 (76.3%) listed an assistant or associate program director, and 199 (94.3%) specified a core faculty member (Table 2). While the survey question had a “public member” option, no respondents indicated as such.Two open-ended questions guided the major focus of the survey: (Q8) “Indicate any challenges you have experienced with Milestones 2.0, or related concerns,” and (Q9) “What changes or resources would you like to see in the Milestones?” Finally, respondents could self-identify under (Q10) “Enter your email and full name to be notified of future Milestones feedback opportunities,” which produced a list of volunteers for future research projects.Researchers from the Milestones department used emergent, in vivo coding to categorize the open-ended responses describing challenges and providing suggestions. While responses reflected specific and individual experiences, some patterns emerged from similar data during analysis that helped paint a picture of the challenges facing the GME community around Milestones. Guided by a heuristic inquiry methodology, the research team performed the coding in the MAXQDA software (VERBI Software GmbH) to label the open-ended responses and look for the patterns using a constant comparative analysis approach.1 The most prevalent challenges and suggestions are discussed below with exact verbiage lifted from the responses.The first prominent idea uncovered in the response data involved faculty use and interpretation of the Milestones. Particularly, many respondents mentioned faculty not being fully invested in the Milestones as an assessment tool and that faculty may consider the goals of the Milestones as too lofty or unachievable.Another common topic from the data highlighted issues with specific subcompetencies, namely those that measure advocacy or well-being. Some respondents called out a few Milestones that were difficult to observe as well as directly assess, particularly those speaking to system management and the physician role in the health care system. Similarly, there were comments that described the Milestones as being too vague or subjective and too difficult to transfer to the provided scale. One comment stated: “Based on how these are worded many of our residents hit a level 3 at PGY-2 and then stay there despite continuing to improve in their abilities.” (internal medicine)Data also reflected issues with the amount of time required to adequately assess residents and fellows on each Milestones subcomeptency. This may be related to other comments where respondents felt there were too many Milestones to assess or that the Milestones were too long or wordy. “Too many Milestones! Takes way too long to complete them twice a year for all of our residents.” (diagnostic radiology)Some respondents described challenges with transitioning from Milestones 1.0 to 2.0, such as making adjustments to their assessment practices and software tools to reflect the changes. One comment pointed out that it would get easier: “There are some that don't correlate perfectly with the new milestones, so this made tracking resident improvement over time difficult. This should get easier once we use the 2.0 version for all 3 resident years.” (pediatrics)There were also a few respondents who felt as though the Milestones were not applicable to actual clinical practices. For example: “I find the wording of the Milestones 2.0 incredibly hard to apply to real world resident performance.” (internal medicine) and “The verbiage is still extremely convoluted and often does not apply to real life.” (family medicine) Others indicated that the Milestones do not capture the context of the resident's “rotation experience.” (family medicine)Similarly, data from the survey described issues with few opportunities for residents to be assessed or actually observed within specific Milestones contexts. They also spoke of other challenges: “Bias from prior milestone evaluations. Some CCC members may evaluate residents that they have not personally worked with over the past 6 months.” (surgery–surgical oncology) and “Finding ways for residents to achieve, and for us to observe and document, some milestones elements (ex. SBP2, SBP3, PROF1, ICS1), especially those in the upper levels.” (pathology–surgical pathology)There remained some responses that did not fall into a particular pattern or theme but still highlighted concerns worth mentioning:The survey respondents' suggestions were just as varied as their challenges, but some themes were evident. The most prominent area for suggestions centered around the need for clearer applications and more examples for the Milestones levels, either beyond the existing supplemental guidebooks or an updated version of them.Another common thread among the suggested changes was the inclusion of either additional Milestones or a new focus reflected within Milestones. Particularly, these responses spoke to skills development in leadership, teaching, and scholarly activities.Respondents also focused their suggestions on language simplification and applying more realistic language as well as condensing Milestones, making them shorter or even reducing the overall number of them.Another theme identified the need for more faculty development and education that address “goals, changes, and interventions to address failure to progress.” (surgery–acute care surgery) Responses also spoke to the need for evaluation forms that would be easy to use and implement into the learner's record automatically. Some respondents suggested using the Milestones as the assessment tool: “Evaluation tools would be helpful, not just the milestones but also some suggested way to assess. Or just let us use the milestones as assessment tools, because the rubrics are built out well enough now that they could be used as direct assessments.” (anesthesiology)Individual comments spoke to a wide variety of suggestions, such as the following:This survey was developed to collect general, voluntary feedback with only 2 focused questions for challenges and suggestions as an early look informing program evaluation for Milestones. While the data are not representative of any group or subgroup within GME, the broad and organic analysis shows that the GME community feels that the Milestones program should continue its focus on providing a transparent and concise assessment tool that does not overburden faculty, program directors, or the residents and fellows.2-5 The survey results also indicate the need to reiterate the validity of the Milestones assessments in the harmonized competencies as well as in patient care and medical knowledge.6,7 There is evidence supporting more emphasis on developing a concerted effort by faculty and CCCs to utilize the tools and resources that the Milestones program provides. Some survey respondents also expressed the need for Milestones to continue efforts for developing subcompetencies that could integrate with other specialty-specific assessment tools.The Milestones department has already researched and taken steps to provide GME faculty with access to resources, such as guidebooks (Milestones, CCCs, assessment, etc) and faculty development courses,8,9 and currently offers periodic courses to address faculty understanding and usage of the Milestones assessments. The department also has developed supplemental resources and has explained the rigor that goes into developing the validity and reliability of the tools.10 The data also spoke to the importance of keeping current and future resources visible, available, and accessible to the GME community.The ACGME Milestones team would like to thank those who took part in the survey and provided their valuable feedback. The resulting suggestions and conversations will give the Milestones department the foundations to better inform their upcoming evaluation research on the Milestones program. Over the next few years an intensive review of the Milestones will occur. This research will delve into the development process, content, and implementation within programs. Data will be collected through focus groups, interviews, content analyses, and targeted surveys. If you are interested in sharing your opinion, please send an email with your name, specialty, and/or subspecialty to MilestonesQA@acgme.org.

Is there a gender bias in milestones evaluations in general surgery residency training?

ABSTRACTBackground: The Accreditation Council for Graduate Medical Education (ACGME) introduced milestones for Emergency Medicine (EM) in 2012. Clinical Competency Committees (CCC) are tasked with assessing residents on milestones and reporting them to the ACGME. Appropriate workflows for CCCs are not well defined.Objective: Our objective was to compare different approaches to milestone assessment by a CCC, quantify resource requirements for each and to identify the most efficient workflow.Design: Three distinct processes for rendering milestone assessments were compared:Full milestone assessments (FMA) utilizing all available resident assessment data,Ad-hoc milestone assessments (AMA) created by multiple expert educators using their personal assessment of resident performance,Self-assessments (SMA) completed by residents.FMA were selected as the theoretical gold standard. Intraclass correlation coefficients were used to analyze for agreement between different assessment methods. Kendall’s coefficient was used to assess the inter-rater agreement for the AMA.Results: All 13 second-year residents and 7 educational faculty of an urban EM Residency Program participated in the study in 2013. Substantial or better agreement between FMA and AMA was seen for 8 of the 23 total subcompetencies (PC4, PC8, PC9, PC11, MK, PROF2, ICS2, SBP2), and for 1 subcompetency (SBP1) between FMA and SMA. Multiple AMA for individual residents demonstrated substantial or better interobserver agreement in 3 subcompetencies (PC1, PC2, and PROF2). FMA took longer to complete compared to AMA (80.9 vs. 5.3 min, p < 0.001).Conclusions: Using AMA to evaluate residents on the milestones takes significantly less time than FMA. However, AMA and SMA agree with FMA on only 8 and 1 subcompetencies, respectively. An estimated 23.5 h of faculty time are required each month to fulfill the requirement for semiannual reporting for a residency with 42 trainees.

On July 1, 2013, the first group of programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) transitioned into the Next Accreditation System (NAS), and 1 year later, the remaining accredited specialties, the transitional year, and the review of sponsoring institutions will move into the NAS.1 The NAS is an outcomes-based and improvement-focused approach to the accreditation of programs and sponsoring institutions. The intent is to both ensure a level of quality (through assessment of compliance with the accreditation standards) and to promote ongoing improvement in all programs and innovation in high-performing programs.1 The added focus on improvement will be achieved through (1) a Self Study that each program will complete before its 10-year site visit and includes aggregation and analysis of data on its longitudinal improvement effort, and (2) a Self Study site visit during which the program will describe its improvement process and the outcomes achieved. The Self Study site visit will occur approximately 1 year after the program begins its Self Study.The lengthening of site visit intervals in the NAS and the focus on outcomes and continuous improvement call for a new type of site visit, and this new model requires testing. To further inform the Self Study and to increase its relevance to programs, the ACGME envisioned a new focus on programs' aims, combined with an environmental assessment that focuses on strengths, areas for improvement, opportunities, and threats.A key aim in the NAS is to reduce the burden of accreditation, and a recognized source of burden is the preparation of site visit documentation. Thus, the Self Study site visit also calls for a new approach to reporting that provides insight into programs' longitudinal improvement efforts, without that documentation either adding undue burden or affecting the ability of the program to conduct a complete and honest accounting of its strengths and its areas in need of improvement.The Self Study approach is currently being refined, with the first Self Studies for programs in Phase I of the NAS scheduled to begin in 2014, and the first Self Study site visits planned for 2015. In this brief article, we offer early information from the ongoing, conceptual testing of the Self Study site visit.By 2012, the aims and content of the 10-year Self Study and Self Study site visit (box 1) had evolved sufficiently to allow the ACGME to test the new elements of the proposed approach. This included the enhanced focus on program improvement, using the Annual Program Evaluation, and the new focus on program aims and an assessment of programs' performance and opportunities and threats in their local environment.In appreciation of the significant development the Self Study process and the new type of site visit would require, ACGME leadership thought it would be beneficial to test the emerging concepts and approaches in a sample of Self Study test visits, beginning in late 2012. It was clear that a “real” Self Study site visit would have required the programs to have completed a self-assessment and that the test visits would test only selected elements of the planned approach.To date, site visits at 3 institutions involving internal medicine and plastic and vascular surgery core programs and subspecialty programs in those areas have been completed. Test visits involved volunteer institutions, and the findings were not shared with Residency Review Committees. All test site visit programs were visited by a team, as is planned for the actual Self Study site visits. The individuals interviewed and the time commitment for participants on the day of the site visit were comparable to a “regular” site visit. The ACGME sought to ensure that preparation before a site visit would be minimal, and the site visit was limited to asking the residents or fellows in the test programs to complete a consensus list of program strengths and areas for improvement.The test visit entailed a debriefing with site visitors and program participants and produced reports for internal sharing and comprehensive field notes. In contrast to the more general compliance focus that had characterized earlier tests of Program Information Form-less site visits,2 these visits tested selected new elements planned for future Self Study visits. They included a focus onItems 2 through 4 above constituted an initial test of new dimensions the ACGME envisioned as part of a future NAS 10-year Self Study site visit. One of the key information items for this portion of the visit was the Annual Program Evaluation. To continue to preserve the ability of programs to conduct a frank assessment of areas for improvement, the test visits maintained the confidentiality of the Annual Program Evaluation, and the team asked the program director to discuss program improvement, articulating successes achieved and areas still being worked on, based on the several recent Annual Program Evaluations. The ACGME envisions that a similar approach will be used for actual Self Study visits.Although the sample is small, the test site visits to date have confirmed the feasibility and “assess-ability” of the added dimensions of a Self Study visit. Programs articulated “aims” without difficulty, although, other than efforts to address lower than desirable board performance, offering general information on preparing residents for academic careers, and in 1 instance, starting a primary care track, respondents were less clear about their ongoing activities to enhance performance relevant to these aims.Discussion during the assessment of strengths, areas for improvement, opportunities, and threats focused primarily on national threats, particularly changes in federal support for graduate medical education (GME). Local opportunities and threats were less readily identified in the conversations with program leadership. This suggests that these new site visit dimensions would benefit from ongoing assessment over time, and it is expected that these areas would be more developed in a “real” Self Study, which would have been preceded by a formal Self Study process.The early test site visits also highlighted several areas of emerging focus, some likely prompted by the increased emphasis on educational outcomes in the NAS. During the site visits, programs discussed new modes of evaluation on the Milestones; on initiatives to restructure their didactic programs, focusing on scheduling and delivery models to maximize access and participation; and on measuring the effect of the didactic offerings. Preparation for the board examination, as the most visible element of outcomes focus in the current system, constituted a major focus of the discussion of educational outcomes across programs at the programs sites visited to date. Another focal area likely resulted from new roles for faculty in the NAS, particularly professional development for assessing performance on the milestones and participation on Clinical Competency Committees. In addition to faculty development content, discussions addressed approaches to encourage participation, including scheduling strategies and financial and other incentives, such as academic promotion and recognition.Finally, the visits highlighted some beneficial approaches for tracking data on longitudinal improvement, using simple, common institutional formats that allowed programs to record and report their improvements and areas still being worked on. This longitudinal tracking of improvement will gain in importance in the NAS, as much of the focus of the Self Study visit will be on programs' effectiveness in making ongoing improvements.The early tests contributed significantly to the development and refinements of the program Self Study approach and the associated site visits envisioned for the NAS. This work will continue with a focus on (1) offering guidance to programs about effective and efficient approaches for longitudinal improvement, including sessions at the ACGME Annual Educational Conference; (2) developing a report that aggregates NAS performance data the ACGME would provide to programs at the “kick-off” of their Self Study effort, approximately 1 year before the scheduled site visit; and (3) formulating a succinct document for programs to complete in preparation for a Self Study site visit.During the early test visits, it became clear that a truly meaningful Self Study visit would require the prior completion of a Self Study process. The concept is widely used in educational accreditation, yet it remains one of the more debated elements. The ideal approach for an NAS Self Study would produce a self-assessment that aids in the acquisition of knowledge about the program and identifies areas of real, tangible improvements.3 At the same time, organizational and programmatic Self Studies, although valuable from an improvement perspective, carry significant time, financial, and opportunity costs.4 It is clear that, given the smaller size of residency programs or even a core programs and its associated subspecialty programs compared with universities or medical schools and the resource constraints in the GME environment, the Self Study approach will need to be adapted and streamlined for use in the NAS. It also became apparent that the process would need to address scheduling challenges for programs, particularly when core programs and their subspecialty programs are seen as a group, as is planned in the NAS. The rationale and anticipated benefits for having a collective Self Study and Self Study site visit for core programs and their associated subspecialty programs are shown in box 2.The exploration of elements of the Self Study and the associated visit confirmed the feasibility of the approach and acceptance by program and institutional participants, although the sample was limited and the use of the information for internal improvement may have produced a more receptive and less critical audience. Future challenges and areas to further develop and refine the site visit approach in the NAS will need to deal with efforts to assess programs' ongoing focus on improvement while ensuring the confidentiality of the data underlying that process.Additional site visits are planned, with a particular focus on tapping into programs' ongoing improvement efforts, and how those efforts could be documented without impinging on their ability to conduct frank and forthright assessments of strengths and areas for improvement. Future work will also explore formats that offer added value to programs, such as a Self Study or site visit that yields valuable insight for programs in key areas, particularly recognition of opportunities and threats within their environment.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

ACGME's Early Adaptation to the COVID-19 Pandemic: Principles and Lessons Learned.

Before exploring the projected future of Sponsoring Institutions (SIs), it is important to understand their origins and development. Graduate medical education (GME) has relied on diverse learning communities to educate residents for unsupervised practice. As 1 such learning community, the SI has contributed to the education of residents and fellows by ensuring the provision of support systems, resources, and administrative structures. Through their oversight of GME, SIs have fostered clinical learning and working environments in which residents and fellows achieve educational milestones that indicate their ability to provide high-quality, safe patient care upon completion of their programs and throughout their professional lives. What follows is a brief history of institutional accreditation by the Accreditation Council for Graduate Medical Education (ACGME), the process by which we have come to recognize the SI as that community necessary to form physicians.In 1982, the newly formed ACGME published its “Essentials of Accredited Residencies in Graduate Medical Education,” which included a set of “General Requirements” that established standards and responsibilities for GME programs. Part I of these General Requirements began with the declaration that “[p]rograms in graduate medical education are sponsored by institutions engaged in providing medical care and health services. The principal institutions for graduate medical education are hospitals.”2 In Section 1, “Responsibility of Institutions,” the ACGME outlined standards that remain at the core of the essential relationships and processes expected of every institution that seeks to educate residents: commitment to sponsorship agreed upon by both faculty and administration; distribution of resources for educational purposes; establishment of institutional policies by which program directors exercised various responsibilities for their respective programs; periodic analyses of each program's effectiveness in meeting its goals and objectives; provision of facilities and resources to support education; and maintenance of hospital accreditation.3 It is important to note that, at the time, these standards were not new to GME; they were based on well-established groundwork laid by the ACGME's predecessors in program accreditation, namely, the American College of Surgeons, the American Board of Internal Medicine, the American Medical Association's Council on Medical Education, and the Liaison Committee on Graduate Medical Education. Each of these organizations had previously acknowledged the hospital's role as GME sponsor and had included some variation of these expectations in their respective requirements.To fulfill its accreditation mission, the ACGME delegated authority to specialty-specific Review Committees (RCs) that evaluated individual residency programs for compliance with both the General Requirements and specialty-specific program requirements. Although the ACGME's General Requirements acknowledged SIs and the role they played in GME, attention to how these institutions fulfilled their responsibilities was initially provided in the context of individual residency program review conducted by RCs. While these specialty-specific RCs made accreditation decisions independently of each other, each Committee's activity was reviewed periodically by the ACGME through its Monitoring Committee. As a result, in these earliest years of ACGME's development as an accrediting body, there was no process by which the institutions were reviewed separately from specialty programs, nor was there consideration of how a single institution demonstrated compliance with its responsibilities as specified in the General Requirements for all the programs it sponsored.Gradually, the ACGME and its RCs began to observe that noncompliance with program requirements often had, at least, some correspondence to areas of institutional responsibility identified in the General Requirements. In 1992, as a result of this increasing awareness, the ACGME approved a separate section of the General Requirements identified specifically as Institutional Requirements, along with a process for institutional review that was initially administered through the ACGME's Monitoring Committee. These Institutional Requirements elaborated on the standards that had been outlined in the original General Requirements. This initial institutional review process included a separate institutional site visit. On the basis of the site visit report and information provided in an Institutional Review Document, an institutional review administrator determined whether the SI had adequately documented and implemented policies in compliance with the General Requirements. The administrator then prepared a list of the institutions judged to be in compliance with the General Requirements. The Monitoring Committee reviewed documentation only from a representative sample of these institutions. Only 2 decision options were available. Institutions that received a favorable decision were given 5 years until their next review; institutions receiving an unfavorable decision were assigned a time frame for their next review, not to exceed 3 years. Three consecutive unfavorable decisions for an institution resulted in a loss of its ability to sponsor residency and fellowship programs.During this period, the growth of medicine was characterized in part by new specialties and subspecialties approved for certification by various specialty boards and for accreditation by the ACGME. The result was an increased workload for the RCs, due to the number of applications by academic medical centers and community teaching hospitals for new specialty and subspecialty programs. While members of RCs were expert evaluators of programs in their specialty, lengthier review agendas, coupled with the added need to understand complex issues from an institutional perspective, underscored the growing need for a separate RC to review institutions. It also became clear to the ACGME that a cadre of individuals familiar with GME from an institutional perspective, and with knowledge of the General Requirements, should provide an expert review for institutional compliance.In response to these internal and external factors in the health care delivery environment, in GME itself, and within its own internal structure, the ACGME officially constituted the Institutional Review Committee (IRC) in 1995. The IRC was structured in the same manner as the specialty-specific RCs, except with regard to appointment of its members. In contrast to the RCs, the IRC had no formal specialty board or academic medical society of individuals with leadership responsibility for GME at the institutional level from which to draw its membership. As a result, the appointing organization for the IRC became the ACGME itself, with the original members drawn from GME leadership across the country.The only options available to the newly formed IRC with regard to the review of SIs remained the previously used statuses of favorable and unfavorable. The outcomes of these reviews were not, strictly speaking, accreditation decisions, although common usage in the field often referred to them as such. Loss of the ability to sponsor accredited programs remained as the action after 3 unfavorable decisions were given to an institution. Although such a decision was never made by the IRC, it nonetheless highlighted the growing importance of institutional review. While the seeds of accreditation were sown, the time for official institutional accreditation would wait for nearly a decade.As health care delivery and funding became more complex through the 1990s, the Institutional Requirements became more detailed. Experience with the institutional review led the ACGME and the IRC to identify with greater specificity the various elements and relationships necessary for maintaining an effective SI in this environment. Managing GME relationships across the institution was an effort that assumed progressively more responsibility and importance. For example, in 1993, the standard calling for each SI to have a Graduate Medical Education Committee (GMEC) was added to the Institutional Requirements; in 1997, the appointment at each SI of a Designated Institutional Official (DIO) as chief administrator for GME was approved. The DIO and GMEC were charged with oversight of the SI's GME efforts. The DIO became the “designated” leader with whom the ACGME could communicate information that affected all programs in the SI, starting with the basic responsibility of receiving the ACGME's annual invoice for accreditation services. Eventually, the DIO became the individual in the SI who was recognized as having the authority to act on behalf of the entire ACGME-accredited enterprise. The DIO had responsibility for accreditation matters, and also assumed many locally defined duties commonly associated with management and supervisory oversight in a complex organization. The Institutional Requirements began to codify the relationships needed at the institutional level for support of the educational environment, and the formal structures that comprised the GME local learning community took shape.In the mid- to late 1990s, the ACGME underwent its first intensive strategic planning process. A central component of the resulting plan, which eventually became known as the Outcome Project, was initiated in 1999 with a grant from the Robert Wood Johnson Foundation. The central focus of this effort was the identification of 6 general competencies as organizing principles for specialty and subspecialty curricula. ACGME-accredited programs were expected to identify educational outcomes for residents and to evaluate their achievement based on the general competencies that were eventually codified into all specialty and subspecialty program requirements. During this period, the educational leadership role of the SI's DIO and GMEC became even more apparent to the ACGME, the IRC, and SIs themselves. The need for central oversight of program curricula and resident evaluation processes emerged as a critical institutional responsibility for maintaining overall educational effectiveness within each program and across the institution.After 10 years' experience observing the maturation of the institutional review process and monitoring the work of the IRC, the ACGME Board of Directors delegated full accreditation authority to the IRC in 2005. The IRC acted with this authority to accredit all SIs having 2 or more accredited programs. With this change, the IRC maintained the same structure as all RCs, with the exception of its member selection process, which remained open to nominations of individuals from the entire GME community.In 2007, the IRC completed a major revision of the Institutional Requirements in an effort to remove extraneous language and to reflect the increasing recognition of the SI's central role in overseeing GME. As further indication of the growing importance of institutional review, this revision did not occur in a vacuum. DIOs and GMECs had begun to report that areas of programs' noncompliance cited by RCs occasionally overlapped with citations at the institutional level given by the IRC. One reason for such overlap (and sometimes even dissonance or contradiction) occurred because the Institutional and Common Program Requirements had developed independently. Therefore, an ad hoc committee that included representatives from the specialty RCs and the IRC was created to reconcile these 2 sets of requirements. The result was that a major revision of the ACGME Common Program Requirements also occurred in 2007 and included a thorough reconciliation with the Institutional Requirements. This reconciliation initiative revealed how institutional accreditation gradually had come of age within the ACGME accreditation structure and at the grassroots level. Specialty-specific RCs accepted the IRC's expertise in institutional matters and were no longer compelled to cite expectations outlined in the Institutional Requirements.Consistent with the recognition of the importance of institutional review, the IRC chair eventually became a full voting member of the ACGME Council of Review Committee Chairs, thus assuring an opportunity for ongoing dialogue between the specialty RCs and the IRC. The relationship of program and institutional accreditation was forged; it mirrored the expectation of collaboration among specialty and subspecialty programs at the level of the SI, facilitated by the DIO and GMEC.Today, institutional accreditation is an integral component of the ACGME's ongoing strategic initiatives. With approval of its Next Accreditation System in 2011, the ACGME Board of Directors codified the responsibility of SIs for educational outcomes: “The ACGME accredits GME programs and SIs based on the demonstration of continuous oversight of processes and outcomes of education, and substantial compliance with accreditation standards, through the review of annually acquired information.”4 Effective institutional oversight of a single program or 100 programs is the expectation for institutional outcomes under scrutiny by the IRC as it monitors institutions on an annual basis. Recognizing this central role of the SI in monitoring outcomes, the IRC completed an extensive revision of the Institutional Requirements in 2013. This time, the revision focused on a major reordering and simplification of the language so that additional focus would be placed on demonstrable outcomes at the institutional level, even as the processes evident in the original standards remained.Yet another development in institutional accreditation occurred in 2014 when the ACGME, the American Osteopathic Association (AOA), and the American Association of Colleges of Osteopathic Medicine (AACOM) announced their agreement to form a single GME accreditation system in the United States. This decision allows graduates of allopathic and osteopathic medical schools to complete their residency and/or fellowship education in ACGME-accredited programs and to demonstrate achievement of common milestones and competencies. The move toward a single accreditation system means that many AOA-approved institutions and other entities that oversee AOA-approved programs have applied or will apply for ACGME accreditation as SIs. This landmark agreement has resulted in an expanded assortment of governing structures for SIs in which residents and fellows are educated. In addition to commonly encountered institutional models overseen by academic medical centers, community teaching hospitals, medical schools, and single-program institutions, newly ACGME-accredited SIs include osteopathic postdoctoral training institutions (OPTIs), which are educational entities that provide academic oversight and ensure GME resources for hospitals, colleges of osteopathic medicine, clinics, and teaching health sites, sometimes over large geographic areas.In 2012, the ACGME Board of Directors approved the ACGME's Clinical Learning Environment Review (CLER) program, which “provides the profession and the public a broad view of SIs' initiatives to enhance the safety of the learning environment and to determine how residents are engaged in patient safety and quality improvement activities.”5 Although not an accreditation activity, the CLER program emphasizes the institutional relationships necessary to improve health care and population health.Sponsoring Institutions now define themselves in various ways, often departing from the model of the singular hospital originally identified in the earliest standards for accreditation. What has remained consistent throughout the development of the institution's role in GME, however, is that it is the organizing force that affects GME at all levels, and that creates space for relationships that ensure the quality of resident education.Sponsoring Institutions are currently in a state of accelerated evolution in response to major changes underway in the health care system and beyond. The next stage in the maturation of SIs will necessarily benefit from reflecting on where these changes will lead. It will require the collaboration of the entire GME community to understand how SIs can meet the needs of patients by creating physicians who are prepared to practice in 2025. As in the past, it is through these relationships that future SIs will realize our future vision for GME.

PURPOSE Documentation of resident operative performance remains challenging. Although numerous assessment tools exist, they tend to be lengthy and ill-suited for use after every case, leading to inconsistent or insufficient data informing mandatory semiannual resident reviews. For these reasons, we created a simple, online assessment of resident operative skill to facilitate documentation of training activities in compliance with the American Council of Graduate Medical Education’s (ACGME) Next Accreditation System (NAS). METHODS Working with the Johns Hopkins Technology Innovation Center, we developed an online assessment tool: the Operative Entrustability Assessment (OEA). The tool includes a 5-point scale with descriptive anchors, can be applied to any current procedural terminology code, can be mapped to the NAS Milestones, and mirrors the ACGME case log structure. Department-wide use of the OEA started in January 2014, and residents and faculty were instructed to complete OEAs for all plastic surgery residents performing operative procedures with Department of Plastic Surgery attending physicians. Residents initiate assessments, complete a self-assessment, and hand (or e-mail) the OEA to the attending for completion. Attendings assign cases to the most relevant NAS Milestone and grade the resident on his/her ability to perform the case autonomously. In-person evaluations provide opportunities for immediate feedback; scores are available for viewing in real time via the Data Dashboard. Data can be viewed by Milestone category, CPT code, or case. Data can be further sorted by postgraduate year level, specific dates, Milestone, and CPT code. RESULTS From January 21, 2014, to October 31, 2014, 1298 OEAs were completed by 21 residents and available for use by the Clinical Competency Committee. Six residents did not complete any OEAs due to being on off-service or elective rotations (n = 1); on required, nonclinical (eg, research) rotations (n = 3); or PGY-1 residents assigned to general surgery rotations (n = 2) during the time assessed. Clinical Competency Committee resident assessment and documentation took 275 minutes (4.58 hours), averaging 11.5 minutes (range, 2–28 minutes) per resident across all 36 Milestones, including discussion and recording of action items/assignment of learning activities. OEA data were most often used to inform Patient Care Milestones. Comparisons with the operating room case counts per attending physician estimate the OEA completion rate at approximately 60% of all resident-assisted cases; 38% of residents reported increased immediate feedback on operative performance. CONCLUSIONS Implementation of the OEA has greatly benefitted our semiannual and ongoing assessment of >20 residents, reducing evaluation time >5 times from pilot program evaluation times (1 hour per resident). The OEA also informs program mentor and director the meetings and assignment of training activities. Data transparency provides residents with their own progress compared to the pooled mean of their peers, empowering residents to proactively address areas for potential improvement. Completing OEAs immediately following cases provides valuable opportunities for immediate feedback on operative performance. We continue using the OEA and Data Dashboard for resident assessment and plan to make this tool, which has changed the ACGME case log from a passive to an active measure of operative skill, available to other Plastic Surgery Programs in the near future to validate these findings.

Clinical competency committees (CCCs) are now required in graduate medical education. This study examined how residency programs understand and operationalize this mandate for resident performance review. In 2013, the investigators conducted semistructured interviews with 34 residency program directors at five public institutions in California, asking about each institution's CCCs and resident performance review processes. They used conventional content analysis to identify major themes from the verbatim interview transcripts. The purpose of resident performance review at all institutions was oriented toward one of two paradigms: a problem identification model, which predominated; or a developmental model. The problem identification model, which focused on identifying and addressing performance concerns, used performance data such as red-flag alerts and informal information shared with program directors to identify struggling residents.In the developmental model, the timely acquisition and synthesis of data to inform each resident's developmental trajectory was challenging. Participants highly valued CCC members' expertise as educators to corroborate the identification of struggling residents and to enhance credibility of the committee's outcomes. Training in applying the milestones to the CCC's work was minimal.Participants were highly committed to performance review and perceived the current process as adequate for struggling residents but potentially not for others. Institutions orient resident performance review toward problem identification; a developmental approach is uncommon. Clarifying the purpose of resident performance review and employing efficient information systems that synthesize performance data and engage residents and faculty in purposeful feedback discussions could enable the meaningful implementation of milestones-based assessment.

Fellowship Training in Adult Cardiothoracic Anesthesiology: Navigating the New Educational Landscape as a Result of the Coronavirus Crisis

Is Nephrology Fellowship Training on the Right Track?

The Accreditation and Certification System After Next