Detecting atypical alert behavior through statistical process control: Clinical decision support alert frequency visualizations.

P2Y12 inhibitor selection involves numerous factors, including CYP2C19 genetics, since evidence demonstrates reduced clopidogrel efficacy in patients with decreased or no function CYP2C19 variants. In 2020, Indiana University health embedded interruptive clinical decision support (CDS) alerts in the electronic health record (EHR) to recommend alternative P2Y12 inhibitors for patients with decreased CYP2C19 function who had undergone percutaneous coronary intervention (PCI). The objective of this study was to evaluate prescriber response to these CDS alerts. Utilizing alert response data and P2Y12 inhibitor prescriptions from October 2020 to December 2023, we evaluated prescriber response for 362 patients who had a mean of 2.2 (SD: 1.7) CYP2C19-clopidogrel alerts fire post-PCI. Alert recommendations were accepted 24.5% ± 36.8% (mean ± SD) of the time. Alternative (i.e., non-clopidogrel) P2Y12 inhibitors were prescribed for 22.4% ± 36.8% of days during the year following PCI. Alerts were accepted more for CYP2C19 poor metabolizers than for intermediate metabolizers (P = 0.03). Each year after 2020 was associated with a 4.6% increase in the days prescribed alternative P2Y12 inhibitors. In contrast, increasing age was associated with decreased alert acceptance and decreased percentage of days prescribed alternatives, and concomitant oral anticoagulant use was associated with decreased percentage of days prescribed alternatives. Provider clinical judgment was the most common alert override reason, accounting for 68% of clinician responses. Our findings demonstrate that CYP2C19-clopidogrel CDS alerts promoted genotype-guided P2Y12 inhibitor prescribing in some cases (~22% of study days). Future research should better determine prescriber reasons for rejecting alert recommendations and establish best implementation practices to complement CDS alerts.

To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts. We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness. Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001). End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study

ObjectiveWe describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts.Materials and MethodsWe developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program.ResultsPrior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.ConclusionsAt VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.

The primary aim of this performance improvement project was to determine whether the electronic health record implementation of stroke-specific nursing documentation flowsheet templates and clinical decision support alerts improved the nursing documentation of eligible stroke patients in seven stroke-certified emergency departments. Two system enhancements were introduced into the electronic record in an effort to improve nursing documentation: disease-specific documentation flowsheets and clinical decision support alerts. Using a pre-post design, project measures included six stroke management goals as defined by the National Institute of Neurological Disorders and Stroke and three clinical decision support measures based on entry of orders used to trigger documentation reminders for nursing: (1) the National Institutes of Health's Stroke Scale, (2) neurological checks, and (3) dysphagia screening. Data were reviewed 6 months prior (n = 2293) and 6 months following the intervention (n = 2588). Fisher exact test was used for statistical analysis. Statistical significance was found for documentation of five of the six stroke management goals, although effect sizes were small. Customizing flowsheets to meet the needs of nursing workflow showed improvement in the completion of documentation. The effects of the decision support alerts on the completeness of nursing documentation were not statistically significant (likely due to lack of order entry). For example, an order for the National Institutes of Health Stroke Scale was entered only 10.7% of the time, which meant no alert would fire for nursing in the postintervention group. Future work should focus on decision support alerts that trigger reminders for clinicians to place relevant orders for this population.

To compare rates of prescriber acceptance of interruptive and noninterruptive clinical decision support (CDS) alerts regarding potential diminished therapeutic effectiveness and safety risks associated with proton pump inhibitor (PPI) use in carriers of gene variants affecting cytochrome P450 (CYP) isozyme 2C19 metabolism. A retrospective study was conducted at a large rural health system to examine different approaches to improving CDS alert acceptance while minimizing alert fatigue. Manual reviews were conducted to identify alerts regarding CYP2C19 metabolizer status displayed at the time of PPI ordering over 30-day periods before and after the transition from interruptive to noninterruptive CDS alert functionality. A chi-square test was conducted to analyze prescriber acceptance of CDS recommendations by alert modality and type of treatment modification. Overall, interruptive alerts had an acceptance rate of 18.6% (64/344), compared to 8.4% acceptance (30/357 alerts) for noninterruptive alerts (P ≤ 0.0001). Analysis of acceptance criteria -revealed the noninterruptive alert cohort had higher acceptance, as determined by documented medication dose adjustments, than the interruptive alert cohort (53.3% [16/30] and 4.7% [3/64], respectively). The difference in acceptance rates by CDS modality and treatment modification was statistically significant (P ≤ 0.00001). The predominant indication for PPI use was gastroesophageal reflux disease (GERD) in both cohorts. Interruptive alerts that actively influenced workflow had higher acceptance rates than noninterruptive alerts that served an informational purpose without a direct disruption of workflow. The study results suggest the utilization of noninterruptive alerts may be a beneficial tool for prompting clinicians to alter dosing regimens rather than transition to an alternative agent.

BackgroundWe recently showed that the empirical use of a combination of 2 moving average (MA) statistical process control (SPC) charts was highly sensitive and specific for detecting potentially important increases in surgical site infection (SSI) rates. We performed this follow-up study to examine the performance of these same SPC charts when applied to known SSI outbreaks.MethodsWe retrospectively applied 2 MA SPC charts to all 30 SSI outbreaks investigated from 2007 to 2015 in a network of over 50 community hospitals. These outbreaks were detected via routine SSI surveillance activities that occurred in the network. We reviewed prior outbreak investigation documentation to determine the estimated time of outbreak onset and time of traditional surveillance outbreak detection. The first SPC chart utilized procedure-specific, composite SSI data from the hospital network for its baseline; the baseline for the second chart was calculated from SSI data from the outbreak hospital undergoing analysis. Both charts used rolling baseline windows but varied in baseline window size, rolling baseline lag, and MA window size. SPC chart outbreak detection occurred when either chart had a data point above the upper control limit of 1 standard deviation. Time of SPC detection was compared with both time of outbreak onset and time of traditional surveillance detection.ResultsWith the dual chart approach, SPC detected all 30 outbreaks, including detection of 25 outbreaks (83%) prior to their estimated onset (Figure 1). SPC detection occurred a median of 16 months (interquartile range, 12–21 months) prior to the date of traditional outbreak detection, which never occurred prior to outbreak onset. Both individual SPC charts exhibited at least 90% sensitivity in outbreak detection, but the dual chart approach showed superior sensitivity and speed of detection (Figure 2).ConclusionA strategy that employed optimized, dual MA SPC charts retrospectively detected all SSI outbreaks that occurred over 9 years in a network of community hospitals. SPC outbreak detection occurred earlier than traditional surveillance detection. These optimized SPC charts merit prospective study to evaluate their ability to promote early detection of SSI clusters in real-world scenarios.DisclosuresAll Authors: No reported Disclosures.

We analyzed blood utilization at Stanford Hospital and Clinics after implementing real-time clinical decision support (CDS) and best practice alerts (BPAs) into physician order entry (POE) for blood transfusions. A clinical effectiveness (CE) team developed consensus with a suggested transfusion threshold of a hemoglobin (Hb) level of 7 g/dL, or 8 g/dL for patients with acute coronary syndromes. The CDS was implemented in July 2010 and consisted of an interruptive BPA at POE, a link to relevant literature, and an "acknowledgment reason" for the blood order. The percentage of blood ordered for patients whose most recent Hb level exceeded 8 g/dL ranged at baseline from 57% to 66%; from the education intervention by the CE team August 2009 to July 2010, the percentage decreased to a range of 52% to 56% (p = 0.01); and after implementation of CDS and BPA, by end of December 2010 the percentage of patients transfused outside the guidelines decreased to 35% (p = 0.02) and has subsequently remained below 30%. For the most recent interval, only 27% (767 of 2890) of transfusions occurred in patients outside guidelines. Comparing 2009 to 2012, despite an increase in annual case mix index from 1.952 to 2.026, total red blood cell (RBC) transfusions decreased by 7186 units, or 24%. The estimated net savings for RBC units (at $225/unit) in purchase costs for 2012 compared to 2009 was $1,616,750. Real-time CDS has significantly improved blood utilization. This system of concurrent review can be used by health care institutions, quality departments, and transfusion services to reduce blood transfusions.

ObjectiveThis study aimed to examine user actions within a clinical decision support (CDS) alert addressing hypertension (HTN) in chronic kidney disease (CKD).MethodsA pragmatic randomized controlled trial of a CDS alert for primary care patients with CKD and uncontrolled blood pressure included prechecked default orders for medication initiation or titration, basic metabolic panel (BMP), and nephrology electronic consult (e-consult). We examined each type of action and calculated percentages of placed and signed orders for subgroups of firings.ResultsThere were firings for medication initiation (813) and medication titration (430), and every firing also included orders for nephrology e-consult (1,243) and BMP (1,243). High rates of override (59.6%) and deferral (14.6%) were observed, and CDS-recommended orders were only signed about one-third of the time from within the alert. The percentage of orders that were signed after being placed within the alert was higher for medication initiation than for medication titration (33 vs. 12.0% for angiotensin-converting enzyme inhibitors [ACEi] and 38.8 vs. 14% for angiotensin II receptor blockers [ARBs]). Findings suggest that users are hesitant to commit to immediate action within the alert.ConclusionEvaluating user interaction within alerts reveals nuances in physician preferences and workflow that should inform CDS alert design. This study is registered with the Clinicaltrials.gov Trial Registration (identifier: NCT03679247).

We performed a quality improvement project (31 clinics; July 2021 to October 2023) to increase human papillomavirus vaccination initiation frequency and decrease gender disparity in children aged 9 and 10 years. The 11 process changes included electronic health record clinical decision support (CDS) tools for providers, staff, and parents and medical assistant participation. In phase 1 (preparation), initiation frequency was lower in boys (250 of 1688 visits, 15%) than girls (289 of 1549 visits, 19%; P = .003). In phase 2 (CDS alerts; recommended initiation age lowered from 11 to 9 years), initiation frequency was increased and similar between boys (906 of 1847 visits, 49%) and girls (867 of 1740 visits, 50%; P = .64). In phase 3 (patient portal outreach), initiation frequency was increased further for boys and girls. The multifaceted intervention, including CDS tools and lowering the initiation age, was associated with increased initiation frequency and decreased gender disparity.

Results of a multicentre randomised controlled trial of statistical process control charts and structured diagnostic tools to reduce ward-acquired meticillin-resistant Staphylococcus aureus: the CHART Project

<h3>Background</h3> Traditional strategies for surveillance of surgical site infections (SSI) have multiple limitations, including delayed and incomplete outbreak detection. Statistical process control (SPC) methods address these deficiencies by combining longitudinal...

Usability evaluation of pharmacogenomics clinical decision support aids and clinical knowledge resources in a computerized provider order entry system: A mixed methods approach

Pharmacogenomic (PGx)-guided therapy has demonstrated improved medication-related outcomes and reduced healthcare costs. Interruptive clinical decision support (CDS) alerts are crucial in providing guidance to providers in PGx-informed clinical decision making. However, the integration of comprehensive PGx information into clinical practice remains challenging. Epic’s Genomics Module, featuring Genomic-Indicators (Gen-Ind) aims to address this gap by providing clinicians with a customizable PGx profile within the electronic health record (EHR). We aimed to evaluate the usability of the PGx patient profile (Gen-Ind) implemented through Epic’s Genomic Module at UF Health, providing actionable feedback for improvement and standardization for real-world PGx-CDS implementation. We conducted usability evaluation sessions with ten prescribers at UF Health who had seen at least one PGx Our Practice Advisory (OPA) implemented at UF Health. Participants completed tasks within a test environment, including first and second attempts to navigate to the Genomic-Indicators and exploring its features. Quantitative data were collected through demographic surveys, assessment of task completion times, and a Post-Study System Usability Questionnaire (PSSUQ). Qualitative data was gathered through think-aloud sessions and debrief interviews on which thematic analysis was performed. Quantitative analyses revealed significant improvement in navigation efficiency between first and second attempts (p = 0.004). The PSSUQ indicated positive evaluation for system usefulness and both information and interface quality. Inductive thematic analysis identified five main themes: navigation, workflow integration, visibility of key features, content quality, and suggestions for optimization and alert systems. The Genomic-Indicators may support clinical decision-making and were generally well received by users. Continued refinements in interface design, customization options, and integration of notification to alert clinicians to the presence of the Genomic-Indicators could further improve its utility and clinical value. These findings provide valuable insights into approaches for improving the user experience and clinical relevance of PGx-CDS tools based on clinician feedback.

The presence of autocorrelation in time series data significantly affects the performance of statistical process control charts. For process data collected in real time, consecutive observations result in serial dependence that violates the assumption of sample independence. This paper focuses on the improvement of statistical process control (SPC) charts of individual observations when they are designed to monitor stationary processes in the presence of data autocorrelation. Three widely used SPC tools, Shewhart chart, Hotelling's T2 chart, and PCA‐based SPC charts, are chosen to demonstrate the proposed approach. There has been a modified version of SPC chart shown to be effective in accommodating the autocorrelation behavior, relying on a model‐based estimation of chart parameters. Process model estimation, however, is a challenging task, especially for complex systems. In this paper, a model‐free monitoring approach is proposed for SPC charts of individual observations, aimed at reducing the effect of autocorrelation on chart performance. It relies on a skipping strategy that creates sample bins with little or no autocorrelation. A single skipping chart and a combined skipping chart (CSC) are established, and a parameter sensitivity study is performed through simulations. In the case studies, the advantages of the CSC are noted as being a model‐free approach and having a performance consistent with that of the benchmark modified chart.