Integration of Continuous Quality Improvement Methods Into Annual Program and Institutional Evaluation.

On July 1, 2013, the first group of programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) transitioned into the Next Accreditation System (NAS), and 1 year later, the remaining accredited specialties, the transitional year, and the review of sponsoring institutions will move into the NAS.1 The NAS is an outcomes-based and improvement-focused approach to the accreditation of programs and sponsoring institutions. The intent is to both ensure a level of quality (through assessment of compliance with the accreditation standards) and to promote ongoing improvement in all programs and innovation in high-performing programs.1 The added focus on improvement will be achieved through (1) a Self Study that each program will complete before its 10-year site visit and includes aggregation and analysis of data on its longitudinal improvement effort, and (2) a Self Study site visit during which the program will describe its improvement process and the outcomes achieved. The Self Study site visit will occur approximately 1 year after the program begins its Self Study.The lengthening of site visit intervals in the NAS and the focus on outcomes and continuous improvement call for a new type of site visit, and this new model requires testing. To further inform the Self Study and to increase its relevance to programs, the ACGME envisioned a new focus on programs' aims, combined with an environmental assessment that focuses on strengths, areas for improvement, opportunities, and threats.A key aim in the NAS is to reduce the burden of accreditation, and a recognized source of burden is the preparation of site visit documentation. Thus, the Self Study site visit also calls for a new approach to reporting that provides insight into programs' longitudinal improvement efforts, without that documentation either adding undue burden or affecting the ability of the program to conduct a complete and honest accounting of its strengths and its areas in need of improvement.The Self Study approach is currently being refined, with the first Self Studies for programs in Phase I of the NAS scheduled to begin in 2014, and the first Self Study site visits planned for 2015. In this brief article, we offer early information from the ongoing, conceptual testing of the Self Study site visit.By 2012, the aims and content of the 10-year Self Study and Self Study site visit (box 1) had evolved sufficiently to allow the ACGME to test the new elements of the proposed approach. This included the enhanced focus on program improvement, using the Annual Program Evaluation, and the new focus on program aims and an assessment of programs' performance and opportunities and threats in their local environment.In appreciation of the significant development the Self Study process and the new type of site visit would require, ACGME leadership thought it would be beneficial to test the emerging concepts and approaches in a sample of Self Study test visits, beginning in late 2012. It was clear that a “real” Self Study site visit would have required the programs to have completed a self-assessment and that the test visits would test only selected elements of the planned approach.To date, site visits at 3 institutions involving internal medicine and plastic and vascular surgery core programs and subspecialty programs in those areas have been completed. Test visits involved volunteer institutions, and the findings were not shared with Residency Review Committees. All test site visit programs were visited by a team, as is planned for the actual Self Study site visits. The individuals interviewed and the time commitment for participants on the day of the site visit were comparable to a “regular” site visit. The ACGME sought to ensure that preparation before a site visit would be minimal, and the site visit was limited to asking the residents or fellows in the test programs to complete a consensus list of program strengths and areas for improvement.The test visit entailed a debriefing with site visitors and program participants and produced reports for internal sharing and comprehensive field notes. In contrast to the more general compliance focus that had characterized earlier tests of Program Information Form-less site visits,2 these visits tested selected new elements planned for future Self Study visits. They included a focus onItems 2 through 4 above constituted an initial test of new dimensions the ACGME envisioned as part of a future NAS 10-year Self Study site visit. One of the key information items for this portion of the visit was the Annual Program Evaluation. To continue to preserve the ability of programs to conduct a frank assessment of areas for improvement, the test visits maintained the confidentiality of the Annual Program Evaluation, and the team asked the program director to discuss program improvement, articulating successes achieved and areas still being worked on, based on the several recent Annual Program Evaluations. The ACGME envisions that a similar approach will be used for actual Self Study visits.Although the sample is small, the test site visits to date have confirmed the feasibility and “assess-ability” of the added dimensions of a Self Study visit. Programs articulated “aims” without difficulty, although, other than efforts to address lower than desirable board performance, offering general information on preparing residents for academic careers, and in 1 instance, starting a primary care track, respondents were less clear about their ongoing activities to enhance performance relevant to these aims.Discussion during the assessment of strengths, areas for improvement, opportunities, and threats focused primarily on national threats, particularly changes in federal support for graduate medical education (GME). Local opportunities and threats were less readily identified in the conversations with program leadership. This suggests that these new site visit dimensions would benefit from ongoing assessment over time, and it is expected that these areas would be more developed in a “real” Self Study, which would have been preceded by a formal Self Study process.The early test site visits also highlighted several areas of emerging focus, some likely prompted by the increased emphasis on educational outcomes in the NAS. During the site visits, programs discussed new modes of evaluation on the Milestones; on initiatives to restructure their didactic programs, focusing on scheduling and delivery models to maximize access and participation; and on measuring the effect of the didactic offerings. Preparation for the board examination, as the most visible element of outcomes focus in the current system, constituted a major focus of the discussion of educational outcomes across programs at the programs sites visited to date. Another focal area likely resulted from new roles for faculty in the NAS, particularly professional development for assessing performance on the milestones and participation on Clinical Competency Committees. In addition to faculty development content, discussions addressed approaches to encourage participation, including scheduling strategies and financial and other incentives, such as academic promotion and recognition.Finally, the visits highlighted some beneficial approaches for tracking data on longitudinal improvement, using simple, common institutional formats that allowed programs to record and report their improvements and areas still being worked on. This longitudinal tracking of improvement will gain in importance in the NAS, as much of the focus of the Self Study visit will be on programs' effectiveness in making ongoing improvements.The early tests contributed significantly to the development and refinements of the program Self Study approach and the associated site visits envisioned for the NAS. This work will continue with a focus on (1) offering guidance to programs about effective and efficient approaches for longitudinal improvement, including sessions at the ACGME Annual Educational Conference; (2) developing a report that aggregates NAS performance data the ACGME would provide to programs at the “kick-off” of their Self Study effort, approximately 1 year before the scheduled site visit; and (3) formulating a succinct document for programs to complete in preparation for a Self Study site visit.During the early test visits, it became clear that a truly meaningful Self Study visit would require the prior completion of a Self Study process. The concept is widely used in educational accreditation, yet it remains one of the more debated elements. The ideal approach for an NAS Self Study would produce a self-assessment that aids in the acquisition of knowledge about the program and identifies areas of real, tangible improvements.3 At the same time, organizational and programmatic Self Studies, although valuable from an improvement perspective, carry significant time, financial, and opportunity costs.4 It is clear that, given the smaller size of residency programs or even a core programs and its associated subspecialty programs compared with universities or medical schools and the resource constraints in the GME environment, the Self Study approach will need to be adapted and streamlined for use in the NAS. It also became apparent that the process would need to address scheduling challenges for programs, particularly when core programs and their subspecialty programs are seen as a group, as is planned in the NAS. The rationale and anticipated benefits for having a collective Self Study and Self Study site visit for core programs and their associated subspecialty programs are shown in box 2.The exploration of elements of the Self Study and the associated visit confirmed the feasibility of the approach and acceptance by program and institutional participants, although the sample was limited and the use of the information for internal improvement may have produced a more receptive and less critical audience. Future challenges and areas to further develop and refine the site visit approach in the NAS will need to deal with efforts to assess programs' ongoing focus on improvement while ensuring the confidentiality of the data underlying that process.Additional site visits are planned, with a particular focus on tapping into programs' ongoing improvement efforts, and how those efforts could be documented without impinging on their ability to conduct frank and forthright assessments of strengths and areas for improvement. Future work will also explore formats that offer added value to programs, such as a Self Study or site visit that yields valuable insight for programs in key areas, particularly recognition of opportunities and threats within their environment.

Medical schools provide the foundation for a physician’s growth and lifelong learning. They also require a large share of government resources. As such, they should seek opportunities to maintain trust from the public, their students, faculty, universities, regulatory colleges, and each other. The accreditation of medical schools attempts to assure stakeholders that the educational process conforms to appropriate standards and thus can be trusted. However, accreditation processes are poorly understood and the basis for accrediting authorities’ decisions are often opaque.We propose that increasing transparency in accreditation could enhance trust in the institutions that produce society’s physicians. While public reporting of accreditation results has been established in other jurisdictions, such as Australia and the United Kingdom, North American accrediting bodies have not yet embraced this more transparent approach. Public reporting can enhance public trust and engagement, hold medical schools accountable for continuous quality improvement, and can catalyze a culture of collaboration within the broader medical education ecosystem. Inviting patients and the public to peer into one of the most formative and fundamental parts of their physicians’ professional training is a powerful tool for stakeholder and public engagement that the North American medical education community at large has yet to use.

Creating a Culture of Continuous Quality Improvement: A Pilot of a Quality Team Huddle to Move Knowledge into Action within a Radiotherapy Treatment Program

Objective To evaluate the effect of continuous quality improvement in reducing the incidence of peritonitis in rural patients undertaking peritoneal dialysis. Methods The clinical data of rural peritonitis patients treated at our center from January, 2012 to December, 2013 were retrospectively analyzed and found that paying no attention on aseptic operation was a major cause of peritonitis; and correspondingly, planning, implementation, checking, and correcting the scientific program for continuous quality improvement were taken and measures for peritonitis risk factors and working process were improved. The incidences of peritonitis were compared between before and after the implementation. Results 136 patients undertaking peritoneal dialysis at our center were followed up. The total incidence of peritonitis was 1 time / 19.14 patients month; the incidence of peritonitis of rural patients was 1 time / 17.45 patients month. After the implementation of continuous quality improvement, the incidence of peritonitis of rural patients at our center decreased from 1 time/17.45 patients month to 1 time/ 31.45 patients month. Conclusion Continuous quality improvement of scientific management method can effectively reduce the incidence of peritonitis in rural patients undertaking peritoneal dialysis. Key words: Peritoneal dialysis; Countryside; Peritonitis; Continuous quality improvement

Keywords: percutaneous coronary intervention; structural heart disease intervention; congential heart disease; adults; congenital heart disease; pediatrics

Project management for successful implementation of continuous quality improvement

BackgroundClinical quality registries (CQR) systematically monitor and provide feedback on the appropriateness and effectiveness of health care within specific clinical domains, collecting data about medical care processes and outcomes, and providing benchmarked performance reports to health care providers for the purpose of driving improvements in safety and quality.AimsThe successful development of a learning infrastructure requires the alignment of people, technologies, policies, and processes, brought together by shared needs and a culture of continuous quality improvement. The creation and description of such infrastructure is likely to successfully support the delivery of services critical to continuous quality improvement initiatives.Materials and MethodsWe aimed to describe the construction and impacts of healthcare improvement infrastructure within CQRs.ResultsSocio‐technical infrastructure comprises participants that contribute to the design, evaluation, reporting and dissemination of quality improvement activities: employing effective and timely data acquisition, describing healthcare processes and outcomes; supported by directed policy process and supportive organisations. Review of CQR function identifies positive impacts on healthcare utilisation, improved clinical outcomes and significant improvement in survival supported by cost effective investment.DiscussionQuality improvement frameworks and strategies have been developed to drive CQRs towards international best practice in learning health system structure for data collection and reporting; delivering efficiency and interoperability in data collection and exchange; promoting standardised approaches to CQR design.ConclusionClinical quality registries have demonstrated significant national impacts in quality improvement in high burden disease domains. These registries rely on the construction and resourcing of socio‐technical infrastructures that support quality improvement.

BackgroundThe growing adoption of continuous quality improvement (CQI) initiatives in healthcare has generated a surge in research interest to gain a deeper understanding of CQI. However, comprehensive evidence regarding the diverse facets of CQI in healthcare has been limited. Our review sought to comprehensively grasp the conceptualization and principles of CQI, explore existing models and tools, analyze barriers and facilitators, and investigate its overall impacts.MethodsThis qualitative scoping review was conducted using Arksey and O’Malley’s methodological framework. We searched articles in PubMed, Web of Science, Scopus, and EMBASE databases. In addition, we accessed articles from Google Scholar. We used mixed-method analysis, including qualitative content analysis and quantitative descriptive for quantitative findings to summarize findings and PRISMA extension for scoping reviews (PRISMA-ScR) framework to report the overall works.ResultsA total of 87 articles, which covered 14 CQI models, were included in the review. While 19 tools were used for CQI models and initiatives, Plan-Do-Study/Check-Act cycle was the commonly employed model to understand the CQI implementation process. The main reported purposes of using CQI, as its positive impact, are to improve the structure of the health system (e.g., leadership, health workforce, health technology use, supplies, and costs), enhance healthcare delivery processes and outputs (e.g., care coordination and linkages, satisfaction, accessibility, continuity of care, safety, and efficiency), and improve treatment outcome (reduce morbidity and mortality). The implementation of CQI is not without challenges. There are cultural (i.e., resistance/reluctance to quality-focused culture and fear of blame or punishment), technical, structural (related to organizational structure, processes, and systems), and strategic (inadequate planning and inappropriate goals) related barriers that were commonly reported during the implementation of CQI.ConclusionsImplementing CQI initiatives necessitates thoroughly comprehending key principles such as teamwork and timeline. To effectively address challenges, it’s crucial to identify obstacles and implement optimal interventions proactively. Healthcare professionals and leaders need to be mentally equipped and cognizant of the significant role CQI initiatives play in achieving purposes for quality of care.

The SSM Health Care System is a Catholic, not-for-profit, multihospital health care system with headquarters in St. Louis, Missouri. Sister Mary Jean Ryan, FSM, the President and Chief Executive Officer, has led the implementation of continuous quality improvement (CQI) throughout the system since 1990. This article presents excerpts from a speech Sister Ryan delivered at the system's annual leadership conference in 1990. Sister Ryan delivered this speech at the conclusion of a retreat where the principles of CQI were introduced to the SSM Health Care System leadership. The editors believe this inspiring speech provides insight into the kind of leadership needed to successfully implement quality management.

The goal of postgraduate training is the development of competency necessary for independent practice. To achieve this goal, the Accreditation Council for Graduate Medical Education (ACGME) in 2013 implemented the Milestones assessment framework and mandated the development of clinical competency committees (CCCs), tasked with biannual review of evaluation and assessment data of trainees.1 The CCC, which consists of a chair and committee members, plays a critical role in assisting the program director in decision-making regarding resident or fellow advancement through the training period. Importantly, the committee makeup may span junior and senior faculty and include interprofessional representation, thus providing a spectrum of input.1To date, most of the literature regarding CCCs focuses on optimizing CCC function in terms of workflow and decision-making. Best practices for CCC function have been proposed, including use of varied assessment tools, structured meetings, and diverse members.2,3 Other work has outlined strategies for effective group decision-making necessary for CCC function.4–6 Ongoing faculty development has been proposed as an essential component of CCC function, yet little is known about which particular elements are necessary to optimize organization of meetings, decision-making, and to effectively utilize assessment processes.In order for the CCC to optimally operationalize the Milestones for individual learner development, members of the committee require intentional faculty development. This must span multiple domains important for the CCC process, including knowledge of assessment (and the use of the Milestones), understanding of group decision-making, awareness of how biases interplay with these factors, and the effect of their decisions on patient care (Box). While it remains unknown how often and who should develop such faculty development sessions, we would advocate for at least an annual review of these topics given faculty turnover and need to maintain a shared mental model of best practices. Faculty development remains essential for both new and longstanding members, as participants will likely have different levels of understanding of the assessment process and Milestones themselves.2We therefore aim to provide guidance for specific areas of faculty development for CCC members, including strategies for novel approaches to professional development for the committee. This includes: (1) the purpose and structure of the CCC itself; (2) the strategies, strengths, and weaknesses of the assessments employed by the program; and (3) the development of a shared mental model of the Milestones of the relevant specialty. Beyond this, the CCC chair will also need to understand (4) the best practices of an effective group discussion.7Faculty need to understand the purpose of the CCC meetings, which often employ 1 of 2 strategies: problem identification, where the committee focuses on identifying learners who may require additional remediation, or a developmental approach, where the committee discusses all learners to identify next steps for advancement.3 Members also need to understand the process of decision-making, including who will make the recommendations regarding remediation.In addition to understanding the intent of the Milestones, committee members must also be aware of the program's evaluation process (as related to each Milestone). Knowledge of the individual assessment methods and awareness of the data synthesis for each Milestone will improve a shared mental model and ideally minimize differences in individual interpretation of assessment data.6 This will also allow the committee to identify times when the assessment data is difficult to interpret requiring further group discussion.8Committee members should not only understand each of the relevant Milestones but also have a shared model of what a competent trainee looks like at their institution.9 It is evident that competencies are developed differently in different contexts and at different institutions.10 Therefore, a shared mental model is important for each CCC to minimize variation in how each member interprets the same assessment data.Effective committee groups understand the purpose, structure, and process of their work.4 However, sometimes groups encounter incomplete or otherwise difficult to interpret assessment data; therefore, group discussion is needed to draw a conclusion. The chair of the committee needs to lead the discussion and approach how decisions are made transparently. The chair should also be cognizant of the member composition of the group.4 A homogenous group is unlikely to present different views or opinions, which may be necessary in drawing fair conclusions.Additional faculty development topics for the CCC chair may be helpful in maximizing success of the committee. It is incumbent upon the chair to refine their own ability to synthesize feedback as well as interpret and gauge performance in specialty-specific Milestones. General leadership skills, such as how to effectively lead a committee or run a meeting, remain paramount.11,12 It is important for the CCC chair to adapt their mindset from the frame of educator to that of the “educator-leader.” The CCC chair plays a pivotal role in ensuring that the committee functions through fair and informed decision-making processes based on the data and facts, rather than emotional decision-making.Beyond this unique CCC training for the chair and members, all faculty educators must understand the intersection of biases on the assessment process. It is well-known that biases can manifest in a variety of ways through our traditional mechanisms of assessment. For example, in graduate medical education (GME), this has been highlighted in several settings showing differential Milestone achievement based on trainee gender,13–15 as well as gender differences in narrative evaluation.16 Similarly, standardized patient assessments, which are often included in CCC discussions, have highlighted differential findings by gender, race, and ethnicity.17 Knowledge of such biases and the potential ramifications on Milestones ratings, narrative evaluation, remediation referrals, and progression decisions is imperative for faculty development in the CCC process.Faculty development using bias reduction workshops or training, such as those employing the Implicit Association Test, could represent an avenue to initiate discussions,18 as these training programs have shown to be effective in mitigating biased decision-making in other settings.18–20 The potential quantitative and qualitative differences in assessment should be considered in recurring CCC reviews. Additionally, faculty participants should be versed in the potential impact of system-wide and personal biases on trainee assessments. Beyond an individual training level, CCC structure, including the diversity of CCC members, as well as process measures, including program assessment and trends, should be regularly reviewed for biased tendencies.2There are a multitude of topics important in faculty development for CCC chairs and members and implementation often requires novel solutions. One such approach was recently implemented within the MedStar Health GME Consortium in the Baltimore, MD and Washington, DC region, which includes more than 70 ACGME-accredited programs. The consortium conducted a half-day faculty development retreat specifically for CCC chairs. Topics addressed included a primer on evaluation of competence, a panel discussion of CCC best practices (for varied sizes of programs), small group discussions on CCC decision-making processes, and strategies for running effective meetings. The program also allowed for identification of distinct needs for various CCC stakeholders, as outlined in the Table. These sessions allowed for shared knowledge, highlighted knowledge gaps, and identification of distinct faculty training needs for success of CCCs. Such an approach allowed for sharing of best practices, strategies to overcome real or perceived challenges (such as time constraints to effective discussion in larger scale programs), and dissemination of knowledge throughout the sponsoring institution.Overall, the optimal functioning of a CCC requires dedicated faculty time, education, and investment in faculty development. CCCs form a critical component of GME assessment and play a pivotal role in ensuring readiness for independent practice. Given this critical role, we advocate for an appropriate fraction of protected time (or productivity) from clinical or other administrative duties for CCC chairs and similar key leaders to accomplish the important responsibilities of the role.We have highlighted some overarching themes of essential faculty development for CCC success, but recognize that the needs are varied based on role within the CCC, experience level, and size of the program, among other considerations. It is critical to acknowledge differing faculty development needs across levels of experience of faculty (junior versus senior members), as well as the unique training needs of interprofessional members. Additionally, future scholarship is needed to determine the optimal timing, content, and subsequently the impact of such faculty development on CCC processes. We believe that the continued development of expertise for both the CCC chair and its members remain critical to the success of the Milestones assessment system.

BackgroundImproving quality of care and patient safety practices can strengthen health care delivery systems, improve health sector performance, and accelerate attainment of health-related Sustainability Development Goals. Although quality improvement is now prominent on the health policy agendas of governments in low- and middle-income countries (LMICs), including countries of the Eastern Mediterranean Region (EMR), progress to date has not been optimal. The objective of this study is to comprehensively review existing quality improvement and patient safety policies and strategies in two selected countries of the EMR (Lebanon and Jordan) to determine the extent to which these have been institutionalized within existing health systems.MethodsWe used a mixed methods approach that combined documentation review, stakeholder surveys and key informant interviews. Existing quality improvement and patient safety initiatives were assessed across five components of an analytical framework for assessing health care quality and patient safety: health systems context; national policies and legislation; organizations and institutions; methods, techniques and tools; and health care infrastructure and resources.ResultsBoth Lebanon and Jordan have made important progress in terms of increased attention to quality and accreditation in national health plans and strategies, licensing requirements for health care professionals and organizations (albeit to varying extents), and investments in health information systems. A key deficiency in both countries is the absence of an explicit national policy for quality improvement and patient safety across the health system. Instead, there is a spread of several (disjointed) pieces of legal measures and national plans leading to fragmentation and lack of clear articulation of responsibilities across the entire continuum of care. Moreover, both countries lack national sets of standardized and applicable quality indicators for performance measurement and benchmarking. Importantly, incentive systems that link contractual agreement, regulations, accreditation, and performance indicators are underutilized in Lebanon and absent in Jordan. At the healthcare organizational level, there is a need to instill a culture of continuous quality improvement and promote professional training in quality improvement and patient safety.ConclusionStudy findings highlight the importance of aligning policies, organizations, methods, capacities and resources in order to institutionalize quality improvement and patient safety practices in health systems. Gaps and dysfunctions identified can help inform national deliberations and dialogues among key stakeholders in each study country. Findings can also inform future quality improvement efforts in the EMR and beyond, with a particular emphasis on LMICs.

Central line-associated bloodstream infections (CLABSIs) in NICU result in increased mortality, morbidity, and length of stay. Our NICU experienced an increase in the number of CLABSIs over a 2-year period. We sought to reduce risks for CLABSIs using health care failure mode and effect analysis (HFMEA) by analyzing central line insertion, maintenance, and removal practices. A multidisciplinary team was assembled that included clinicians from nursing, neonatology, surgery, infection prevention, pharmacy, and quality management. Between March and October 2011, the team completed the HFMEA process and implemented action plans that included reeducation, practice changes, auditing, and outcome measures. The HFMEA identified 5 common failure modes that contribute to the development of CLABSIs. These included contamination, suboptimal environment of care, improper documentation and evaluation of central venous catheter dressing integrity, issues with equipment and suppliers, and lack of knowledge. Since implementing the appropriate action plans, the NICU has experienced a significant decrease in CLABSIs from 2.6 to 0.8 CLABSIs per 1000 line days. The process of HFMEA helped reduce the CLABSI rate and reinforce the culture of continuous quality improvement and safety in the NICU.

To the Editor: Program directors or those with administrative roles in psychiatric residency programs know well how we do our very best to keep on top of all the requirements and mandates that are sent by the Accreditation Council of Graduate Medical Education (ACGME). We work diligently on keeping the Program Information Form (PIF) updated, completing evaluations, surveys, and final resident reports. Then, of course, come the competencies. Change can always be a difficult thing to digest, but we do what we can. We conformed to the six competencies—and found new and innovative ways to monitor them in our residents and programs. One ACGME mandate consists of a “program evaluation and improvement plan.” My initial reaction was “What does this mean?” Then I broke it down into two parts: first, we have to evaluate the program, and then if there are concerns, we put a plan in writing to improve upon those concerns. Program directors are constantly making changes, small and not so small, to programs to better the education, training, and learning environment for the residents. A change could be as small as adding a recent article to the reading assignments or as involved as changing a rotation schedule on the basis of concerns that come up during training. These changes are typically made from feedback from residents, staff, nurses, and teaching physicians. It would be very useful to be able to document these program improvement changes in a way that is easily demonstrable to the ACGME during site visits. Most programs look at issues around program improvement and evaluation annually or semiannually. Often committees are formed specially to look at what program changes should be made—another meeting added to already busy schedules. And in practice, do we really make changes to our programs only once or twice a year? Generally, small changes are made continuously. At Michigan State University, we developed a system to record programmatic changes as and when they occur, rather than trying to remember the changes at some distantly scheduled meeting. Noting these changes involves documentation of what was done and who was involved and also feedback from the concerned parties regarding whether the changes accomplished the desired goal. Thus, we put together “C-PIE,” one-page Continuous Program Improvement and Evaluation forms on which we record programmatic changes as they occur (sample form available from the authors upon request). We review completed forms and fill out new forms at our monthly residency education committee meetings. We use these forms to document not only what changes we have made during the course of the year, but also what we did on the basis of the program evaluation forms, faculty evaluation forms, and even resident evaluation forms. Most of those evaluation forms tend to be anonymous, and in those circumstances, we specify on the C-PIE that the concern came from an anonymously completed evaluation. The C-PIE can also be discussed at faculty meetings, resident meetings, or any other meetings. The great advantage of these forms is that we have a continuous, current record of all programmatic changes since the last visit from the Residency Review Committee. The forms remind us of changes made and monitor the effect of the changes. Other advantages are that they are easy to complete, versatile, practical, and more representative of what we as program directors really do on an ongoing basis. Also, the forms document everyone involved in a particular change, can be easily sent to anyone, and can be provided easily to the Residency Review Committee site visitors as proof of actual program evaluation and improvement. In summary, programmatic changes made to improve educational value or address problems can be documented at the time of change. This not only reflects the truth of what occurs but simplifies tracking and recording and makes demonstrating program improvement and change very easy for Residency Review Committee visits.

The Milestones Development team at the Accreditation Council for Graduate Medical Education (ACGME) has periodically surveyed the graduate medical education (GME) community and volunteers, during and after development of the specialty-specific Milestones 2.0, to check with members of each specialty on the implementation of the new Milestones. As we approached the conclusion of Milestones 2.0 development, it was decided to have broader survey dissemination for all specialties. Although still only 10 questions, 2 major changes were made to the quality assurance feedback questionnaire: (1) It solicited comments and impressions from the approximately 26 000-member email distribution list that receives the ACGME e-communications rather than only those who volunteer to be respondents; and (2) It had a more summative focus meant to be the start of the upcoming program evaluation stage of Milestones, rather than soliciting formative input for any further changes to Milestones 2.0. The survey was open for 6 weeks in the fall of 2022 and resulted in 215 responses, which, after clearing the data of collection errors and detectable duplications, produced 211 valid responses from 27 specialties across the GME community. At the time of distribution, email and survey tracking metrics were not available to produce a meaningful response rate. Respondents provided general descriptions of their specialties and specific roles, their program enrollment, and the membership of their Clinical Competency Committees (CCCs). They were then asked open-ended questions that allowed them to describe any challenges with Milestones 2.0 and to make suggestions for the program as a whole. The intent of this 10-item questionnaire was to collect data that will direct conversations on change and help guide this first step, as well as further program evaluation projects that explore the relationships between Milestones and the GME community. The results of the qualitative analysis of this questionnaire are based on those who chose to respond to the ACGME e-communication request for feedback. The subsequent findings should not be generalized to the GME community or be made to represent thoughts or reactions of any subpopulation therein.Of 211 valid responses, roughly half (n=105, 49.8%) identified as program directors (Table 1) from a dropdown list of options (Question [Q]3), and 46 (21.8%) indicated that they were their CCC chair (Q4). The most represented ACGME core specialties (Q1) were internal medicine (n=46, 21.8%), family medicine (n=35, 16.6%), pediatrics (n=24, 11.4%), and surgery (n=17, 8.1%). Additionally, 80 (37.9%) respondents self-identified a subspecialty (Q2) in the open text question following the specialty question.The responses to (Q5) “Indicate the number of residents or fellows you are currently assessing in your program using the Milestones” ranged from 2 to 170. This question was asked to obtain an approximate number of residents and fellows being assessed, and it was beyond the purview of this project to check a respondent's accuracy regarding this data point.When asked to describe the roles within their current CCC membership (Q6), 181 (85.8%) of the respondents listed their committee as having a program director, 162 (76.8%) indicated a program coordinator, 161 (76.3%) listed an assistant or associate program director, and 199 (94.3%) specified a core faculty member (Table 2). While the survey question had a “public member” option, no respondents indicated as such.Two open-ended questions guided the major focus of the survey: (Q8) “Indicate any challenges you have experienced with Milestones 2.0, or related concerns,” and (Q9) “What changes or resources would you like to see in the Milestones?” Finally, respondents could self-identify under (Q10) “Enter your email and full name to be notified of future Milestones feedback opportunities,” which produced a list of volunteers for future research projects.Researchers from the Milestones department used emergent, in vivo coding to categorize the open-ended responses describing challenges and providing suggestions. While responses reflected specific and individual experiences, some patterns emerged from similar data during analysis that helped paint a picture of the challenges facing the GME community around Milestones. Guided by a heuristic inquiry methodology, the research team performed the coding in the MAXQDA software (VERBI Software GmbH) to label the open-ended responses and look for the patterns using a constant comparative analysis approach.1 The most prevalent challenges and suggestions are discussed below with exact verbiage lifted from the responses.The first prominent idea uncovered in the response data involved faculty use and interpretation of the Milestones. Particularly, many respondents mentioned faculty not being fully invested in the Milestones as an assessment tool and that faculty may consider the goals of the Milestones as too lofty or unachievable.Another common topic from the data highlighted issues with specific subcompetencies, namely those that measure advocacy or well-being. Some respondents called out a few Milestones that were difficult to observe as well as directly assess, particularly those speaking to system management and the physician role in the health care system. Similarly, there were comments that described the Milestones as being too vague or subjective and too difficult to transfer to the provided scale. One comment stated: “Based on how these are worded many of our residents hit a level 3 at PGY-2 and then stay there despite continuing to improve in their abilities.” (internal medicine)Data also reflected issues with the amount of time required to adequately assess residents and fellows on each Milestones subcomeptency. This may be related to other comments where respondents felt there were too many Milestones to assess or that the Milestones were too long or wordy. “Too many Milestones! Takes way too long to complete them twice a year for all of our residents.” (diagnostic radiology)Some respondents described challenges with transitioning from Milestones 1.0 to 2.0, such as making adjustments to their assessment practices and software tools to reflect the changes. One comment pointed out that it would get easier: “There are some that don't correlate perfectly with the new milestones, so this made tracking resident improvement over time difficult. This should get easier once we use the 2.0 version for all 3 resident years.” (pediatrics)There were also a few respondents who felt as though the Milestones were not applicable to actual clinical practices. For example: “I find the wording of the Milestones 2.0 incredibly hard to apply to real world resident performance.” (internal medicine) and “The verbiage is still extremely convoluted and often does not apply to real life.” (family medicine) Others indicated that the Milestones do not capture the context of the resident's “rotation experience.” (family medicine)Similarly, data from the survey described issues with few opportunities for residents to be assessed or actually observed within specific Milestones contexts. They also spoke of other challenges: “Bias from prior milestone evaluations. Some CCC members may evaluate residents that they have not personally worked with over the past 6 months.” (surgery–surgical oncology) and “Finding ways for residents to achieve, and for us to observe and document, some milestones elements (ex. SBP2, SBP3, PROF1, ICS1), especially those in the upper levels.” (pathology–surgical pathology)There remained some responses that did not fall into a particular pattern or theme but still highlighted concerns worth mentioning:The survey respondents' suggestions were just as varied as their challenges, but some themes were evident. The most prominent area for suggestions centered around the need for clearer applications and more examples for the Milestones levels, either beyond the existing supplemental guidebooks or an updated version of them.Another common thread among the suggested changes was the inclusion of either additional Milestones or a new focus reflected within Milestones. Particularly, these responses spoke to skills development in leadership, teaching, and scholarly activities.Respondents also focused their suggestions on language simplification and applying more realistic language as well as condensing Milestones, making them shorter or even reducing the overall number of them.Another theme identified the need for more faculty development and education that address “goals, changes, and interventions to address failure to progress.” (surgery–acute care surgery) Responses also spoke to the need for evaluation forms that would be easy to use and implement into the learner's record automatically. Some respondents suggested using the Milestones as the assessment tool: “Evaluation tools would be helpful, not just the milestones but also some suggested way to assess. Or just let us use the milestones as assessment tools, because the rubrics are built out well enough now that they could be used as direct assessments.” (anesthesiology)Individual comments spoke to a wide variety of suggestions, such as the following:This survey was developed to collect general, voluntary feedback with only 2 focused questions for challenges and suggestions as an early look informing program evaluation for Milestones. While the data are not representative of any group or subgroup within GME, the broad and organic analysis shows that the GME community feels that the Milestones program should continue its focus on providing a transparent and concise assessment tool that does not overburden faculty, program directors, or the residents and fellows.2-5 The survey results also indicate the need to reiterate the validity of the Milestones assessments in the harmonized competencies as well as in patient care and medical knowledge.6,7 There is evidence supporting more emphasis on developing a concerted effort by faculty and CCCs to utilize the tools and resources that the Milestones program provides. Some survey respondents also expressed the need for Milestones to continue efforts for developing subcompetencies that could integrate with other specialty-specific assessment tools.The Milestones department has already researched and taken steps to provide GME faculty with access to resources, such as guidebooks (Milestones, CCCs, assessment, etc) and faculty development courses,8,9 and currently offers periodic courses to address faculty understanding and usage of the Milestones assessments. The department also has developed supplemental resources and has explained the rigor that goes into developing the validity and reliability of the tools.10 The data also spoke to the importance of keeping current and future resources visible, available, and accessible to the GME community.The ACGME Milestones team would like to thank those who took part in the survey and provided their valuable feedback. The resulting suggestions and conversations will give the Milestones department the foundations to better inform their upcoming evaluation research on the Milestones program. Over the next few years an intensive review of the Milestones will occur. This research will delve into the development process, content, and implementation within programs. Data will be collected through focus groups, interviews, content analyses, and targeted surveys. If you are interested in sharing your opinion, please send an email with your name, specialty, and/or subspecialty to MilestonesQA@acgme.org.

Impact of a Resident-Driven Quality Improvement Project on Incidental Thyroid Nodule Follow-up-ASingle Center Study.