We used clinical decision support (CDS) to promote compliance with the 21st Century Cures Act's mandate that, with few exceptions, patients be granted timely access to their clinical notes. We conducted an observational analysis of note sharing rates in a large regional health system from February 2, 2021 to October 3, 2023. Throughout the study period, notes were shared with patients by default with the option not to grant note access; starting week 10, clinicians not sharing notes were presented with "hard-stop" CDS requiring selection of an allowable exception reason. Trends were examined with forward step-segmented linear regression. 0.7% of all notes were unshared; rates of unshared notes were highest in pediatrics (4.9%) and psychiatry (2.2%). Rates dropped substantially following hard-stop CDS introduction (downward step of 0.96%; 95% CI -1.17 to -0.024). Despite high portal access (72.6%), few notes were viewed by patients/proxies (17.0%). We found very low overall rates of unshared notes; the significant drop in the rates of unshared notes following the introduction of hard-stop CDS is consistent with prior research showing that hard-stop CDS can be an effective tool. The higher rates of unshared notes in pediatrics and psychiatry likely reflect considerations around sensitive information that are inherent to these fields. CDS effectively promoted note sharing, but patient engagement remained low.

ObjectiveDelirium is a severe condition, often underreported and linked to adverse outcomes such as increased mortality and prolonged hospitalization. Despite its significance, delirium prediction is often hindered by underreporting and inconsistent labeling, highlighting the need for models trained on reliably labeled data (ground truth). This review examines (i) practices for determining labels in delirium prediction models and (ii) how study designs affect label quality, aiming to identify key considerations for improving model reliability.Materials and MethodsA search of Cochrane, PubMed, and IEEE identified 120 studies that met the inclusion criteria.ResultsTo establish the ground truth, 40.8% of studies used routine data, while 42.5% used primary data. The Confusion Assessment Method (CAM) was the most widely used assessment tool (60. 0%). Label and data leakage occurred in 35.0% of studies. High Risk of Bias (RoB) was a recurring issue, with 31.7% of studies lacking sufficient reporting and 36.7% showing inadequate outcome determination. Studies using primary data had lower RoB, whereas those with unclear label sources displayed higher RoB.DiscussionOur findings underscore the importance of careful planning in determining the ground truth frequently neglected in existing studies. To address these challenges, we provide a decision support flowchart to guide the development of more accurate and reliable prediction models.ConclusionThis review uncovers significant variability in labeling methods and discusses how this may affect delirium prediction model reliability. Highlighting the importance of addressing underreporting bias and providing guidance for developing more robust models.

Examine the accuracy of privacy preserving record linkage (PPRL) matches in real world data (RWD). We conducted a systematic literature review to identify articles evaluating PPRL methods from January 1, 2013 to June 15, 2023. Eligible studies included original research reporting quantitative metrics such as precision and recall in health-related data sources. Covidence software was used to manage the review process. Five studies met our inclusion criteria. Tokenization and hash functions were used to hash and encrypt personally identifiable information (PII) including first and last names, dates of birth (DOB), and Social Security Numbers (SSNs) in a variety of RWD. All identified studies utilized deterministic matching. Combinations of tokenized or hashed PII that included "quasi-identifiers" like names and DOBs had consistently high precision (>95%) but lower recall, likely due to misspelled or inconsistently spelled names and name changes. SSN-based combinations demonstrated high precision but variable recall due to incomplete SSN data in RWD. Studies that employed algorithms in which at least one match was identified from a specified set of PII combinations provided high precision and high recall. The systematic review indicates that PPRL methods generally provide highly accurate patient data linkage while maintaining privacy. Researchers should carefully consider the completeness and stability of each PII element selected for PPRL and may want to employ a strategy that allows for patient records to be matched if they meet at least one of several combinations of PII.

The resurgence of syphilis in the United States presents a significant public health challenge. Much of the information needed for syphilis surveillance resides in electronic health records (EHRs). In this manuscript, we describe a surveillance platform for automating the extraction of EHR data, known as SmartChart Suite, and the results from a pilot. The SmartChart Suite framework has been developed in compliance with the HHS Health IT Alignment Policy. The platform's major functionalities are (1) data retrieval; (2) logical evaluation; (3) standardized data storage; and (4) results display. The SmartChart Suite was deployed in September 2023 at the Grady Health System in Atlanta, Georgia. We established a cohort of likely syphilis patients, randomly selected 50 medical records for manual and automated chart review, and analyzed the results. The SmartChart Suite was successfully deployed and integrated with the Epic EHR system at Grady. The overall performance results were precision of 97.6%, recall of 100.0%, and F-Score of 98.8. Automated abstraction of EHR data has significant potential to improve public health surveillance and case investigation processes while reducing the resource burden on health departments and reporters. The SmartChart Suite comprises a flexible open-source solution for registry development and maintenance across a wide spectrum of conditions and use cases. SmartChart Suite demonstrates the potential of automated chart abstraction to support disease surveillance. HHS-compliant open-source tools such as SmartChart Suite can support more efficient human review by providing accurate and relevant data for critical public health activities.

To improve performance of medical entity normalization across many languages, especially when fewer language resources are available compared to English. We propose xMEN, a modular system for cross-lingual (x) medical entity normalization (MEN), accommodating both low- and high-resource scenarios. To account for the scarcity of aliases for many target languages and terminologies, we leverage multilingual aliases via cross-lingual candidate generation. For candidate ranking, we incorporate a trainable cross-encoder (CE) model if annotations for the target task are available. To balance the output of general-purpose candidate generators with subsequent trainable re-rankers, we introduce a novel rank regularization term in the loss function for training CEs. For re-ranking without gold-standard annotations, we introduce multiple new weakly labeled datasets using machine translation and projection of annotations from a high-resource language. xMEN improves the state-of-the-art performance across various benchmark datasets for several European languages. Weakly supervised CEs are effective when no training data is available for the target task. We perform an analysis of normalization errors, revealing that complex entities are still challenging to normalize. New modules and benchmark datasets can be easily integrated in the future. xMEN exhibits strong performance for medical entity normalization in many languages, even when no labeled data and few terminology aliases for the target language are available. To enable reproducible benchmarks in the future, we make the system available as an open-source Python toolkit. The pre-trained models and source code are available online: https://github.com/hpi-dhc/xmen.

In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards. We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC). From August 23, 2021 to March 31, 2022, 35937 medical admissions had 2173 (6.05%) AE compared to 25214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (P <.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively. Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment. Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.

ObjectivesWe propose and validate a domain knowledge-driven classification model for diagnosing post-acute sequelae of SARS-CoV-2 infection (PASC), also known as Long COVID, using Electronic Health Records (EHRs) data.Materials and MethodsWe developed a robust model that incorporates features strongly indicative of PASC or associated with the severity of COVID-19 symptoms as identified in our literature review. The XGBoost tree-based architecture was chosen for its ability to handle class-imbalanced data and its potential for high interpretability. Using the training data provided by the Long COVID Computation Challenge (L3C), which was a sample of the National COVID Cohort Collaborative (N3C), our models were fine-tuned and calibrated to optimize Area Under the Receiver Operating characteristic curve (AUROC) and the F1 score, following best practices for the class-imbalanced N3C data.ResultsOur age-stratified classification model demonstrated strong performance with an average 5-fold cross-validated AUROC of 0.844 and F1 score of 0.539 across the young adult, mid-aged, and older-aged populations in the training data. In an independent testing dataset, which was made available after the challenge was over, we achieved an overall AUROC score of 0.814 and F1 score of 0.545.DiscussionThe results demonstrated the utility of knowledge-driven feature engineering in a sparse EHR data and demographic stratification in model development to diagnose a complex and heterogeneously presenting condition like PASC. The model’s architecture, mirroring natural clinician decision-making processes, contributed to its robustness and interpretability, which are crucial for clinical translatability. Further, the model’s generalizability was evaluated over a new cross-sectional data as provided in the later stages of the L3C challenge.ConclusionThe study proposed and validated the effectiveness of age-stratified, tree-based classification models to diagnose PASC. Our approach highlights the potential of machine learning in addressing the diagnostic challenges posed by the heterogeneity of Long-COVID symptoms.

Among Veterans Health Administration (VHA) patients receiving internet-connected tablets, we examined (1) app usage, (2) clusters of commonly used apps and patient characteristics across clusters, and (3) the relationship between app clusters and mental health services. A survey was distributed to a national sample of VHA patients (n = 5449) within 6 months of tablet receipt. Self-reported app use was clustered with agglomerative hierarchical clustering. Survey data were combined with electronic health record data that included sociodemographics and mental health care utilization during the 6 months following survey completion. Differences in patient characteristics by cluster were examined using chi-square tests. A multivariable model assessed prediction of mental health services across app clusters. Among 1664 patients with complete survey and health care utilization data, the majority (74%) reported using at least 1 app. The most used apps were for video visits (60%) and VA's patient portal (53%). Cluster analysis revealed 5 distinct groups reflecting app usage: Patient Portal (n = 685, 41%), Video Connection (n = 319, 19%), Treatment Companion (n = 166, 10%), Self-Management (n = 85, 5%), and Non-App Group (n = 409, 25%). Cluster characteristics varied, and those in the Patient Portal Group (adjusted odds ratio = 1.64; 95% CI = 1.15-2.32) and Video Connection Group (adjusted odds ratio = 1.52; 95% CI = 1.02-2.26) were more likely to have a mental health visit in the 6 months after survey completion when compared to the Non-App Group (P < .05). Tablet provision supports opportunities to use apps for multiple health management tasks, and app use clusters vary across patient characteristics and health care utilization. Patients most commonly used apps to connect with their care team, and use of some apps and app combinations may improve engagement with mental health services.

Implicit bias perpetuates health care inequities and manifests in patient-provider interactions, particularly nonverbal social cues like dominance. We investigated the use of artificial intelligence (AI) for automated communication assessment and feedback during primary care visits to raise clinician awareness of bias in patient interactions. (1) Assessed the technical performance of our AI models by building a machine-learning pipeline that automatically detects social signals in patient-provider interactions from 145 primary care visits. (2) Engaged 24 clinicians to design usable AI-generated communication feedback for their workflow. (3) Evaluated the impact of our AI-based approach in a prospective cohort of 108 primary care visits. Findings demonstrate the feasibility of AI models to identify social signals, such as dominance, warmth, engagement, and interactivity, in nonverbal patient-provider communication. Although engaged clinicians preferred feedback delivered in personalized dashboards, they found nonverbal cues difficult to interpret, motivating social signals as an alternative feedback mechanism. Impact evaluation demonstrated fairness in all AI models with better generalizability of provider dominance, provider engagement, and patient warmth. Stronger clinician implicit race bias was associated with less provider dominance and warmth. Although clinicians expressed overall interest in our AI approach, they recommended improvements to enhance acceptability, feasibility, and implementation in telehealth and medical education contexts. Findings demonstrate promise for AI-driven communication assessment and feedback systems focused on social signals. Future work should improve the performance of this approach, personalize models, and contextualize feedback, and investigate system implementation in educational workflows. This work exemplifies a systematic, multistage approach for evaluating AI tools designed to raise clinician awareness of implicit bias and promote patient-centered, equitable health care interactions.

Individual-level information about negative life events (NLE) such as bankruptcy, foreclosure, divorce, and criminal arrest might improve the accuracy of machine learning models for suicide risk prediction. Individual-level NLE data is routinely collected by vendors such as Equifax. However, little is known about the acceptability of linking this NLE data to healthcare data. Our objective was to assess preferences for linking external NLE data to healthcare records for suicide prevention. We conducted a discrete choice experiment (DCE) among Kaiser Permanente Washington (KPWA) members. Patient partners assisted in the design and pretesting of the DCE survey. The DCE included 12 choice tasks involving 4 data linking program attributes and 3 levels within each attribute. We estimated latent class conditional logit models to derive preference weights. There were 743 participants. Willingness to link data varied by type of information to be linked, demographic characteristics, and experience with NLE. Overall, 65.1% of people were willing to link data and 34.9% were more private. Trust in KPWA to safeguard data was the strongest predictor of willingness to link data. Most respondents supported linking NLE data for suicide prevention. Contrary to expectations, People of Color and people who reported experience with NLEs were more likely to be willing to link their data. A majority of participants were willing to have their credit and public records data linked to healthcare records provided that conditions are in place to protect privacy and autonomy.