Improving explainability of deep neural network-based electrocardiogram interpretation using variational auto-encoders.

Background Clinicians leverage information from multiple data sources, such as the electrocardiogram (ECG) and echocardiogram to diagnose and manage cardiovascular disease. Deep learning models have shown immense potential to enhance these clinical processes by capturing digital biomarkers of cardiovascular disease from diagnostic modalities. However, the development of multimodal algorithms capable of making joint inferences from multiple modalities, mirroring the clinicians’ decision-making, has remained an unmet need. Purpose We aimed to develop a multimodal foundation model of ECG and echocardiogram as an integrative deep learning approach to evaluating cardiovascular conditions. Methods Twelve-lead ECGs and transthoracic echocardiograms performed at a large academic health system between 2001-2021 were collected. Individuals with paired ECGs and echocardiograms recorded within 3 years were identified and the data was split at the individual level into training, validation, and test sets (80/10/10%). Using paired ECGs and echocardiograms, we developed a deep learning model in a self-supervised setting with contrastive loss and merged embeddings in a joint latent space (Figure 1). We evaluated the performance of this foundation model after fine-tuning for left ventricular systolic dysfunction (LVSD defined as left ventricular ejection fraction of 40% or less), atrial fibrillation, age, and sex as auxiliary tasks. The performance of the multimodal model was compared with those of supervised unimodal ECG-only and echocardiography-only models for each task. Results In total, 4,391,610 videos from 129,424 echocardiography studies paired with 1,094,656 ECGs from 53,876 unique individuals were included (67 ± 14 years of age, 43% female, Figure 1). The multimodal algorithm demonstrated excellent discrimination for disease classification in the test set. In classification of LVSD, for example, it achieved an area under the receiver operating characteristic (AUROC) curve of 0.94, area under precision recall curve (AUPRC) of 0.99 among individuals with LV ejection fraction greater than 40%, and AUPRC of 0.72 among those with LV ejection fraction of 40% or less. The multimodal model consistently outperformed the unimodal algorithms, including the ECG-only and echocardiography-only models in classification of LVSD (AUROC ECG+Echo 0.94, ECG 0.89, Echo 0.89), atrial fibrillation (AUROC ECG+Echo 0.95, ECG, 0.94, Echo 0.75), sex (AUROC ECG+Echo 0.93, ECG 0.87, Echo 0.78) and age (R for variance ECG+Echo 0.75, ECG 0.67, Echo 0.58, Figure 2). Conclusion We have developed a multimodal foundation model for joint inference from ECGs and echocardiograms for diagnosis of cardiovascular conditions. This novel approach outperforms the conventional unimodal models in disease diagnosis and represents an advance in integrating the clinical applications of artificial intelligence.Figure 1 Figure 2

Deep Learning Electrocardiographic Analysis for Detection of Left-Sided Valvular Heart Disease

Funding Acknowledgements Type of funding sources: Public grant(s) – National budget only. Main funding source(s): NWO Rubicon (452019308) Amsterdam Cardiovascular Sciences Background Left Ventricular Ejection Fraction (LVEF) is suboptimal as a sole marker for predicting sudden cardiac death (SCD) or benefit from an ICD. Machine (ML) and deep (DL) learning provide new opportunities for personalised predictions using complex, multi-modal physiological data. Objective We hypothesise that risk stratification for ICD implantation can be improved by ML and DL models that combine clinical variables with time series features from 12-lead electrocardiograms (ECG). Methods We present a multicentre study of 1010 patients with an ischaemic, dilated or non-ischaemic cardiomyopathy and LVEF≤35% implanted with an ICD between 2007 and 2021 for primary prevention of SCD (64.9 ±10.8 years, 73.2% male) in two academic hospitals. For each patient, raw 12-lead, 10-second ECG-recordings obtained <90 days before ICD implantation and clinical details were collected. Supervised ML and DL models were trained and validated using stratified k-fold cross-validation (5 repeats, k=10) on a development cohort (n=550) from Hospital A. We used this model to predict ICD non-benefit defined as mortality without prior appropriate ICD-therapy on an external dataset from Hospital B (n=460). Kaplan-Meier survival analysis stratified by high vs. low predicted probability using Youden's J statistic as cut-off was performed. Results At 3-year follow-up, 16.0% of patients had died of whom 72.8% met the criteria for ICD non-benefit. Extreme gradient boosting models identified subjects with ICD non-benefit with an area under the receiver operator curve (AUROC) of 0.897 ±0.05 during internal validation (Figure 1, solid line). In the external cohort, the AUROC was 0.793 (95% CI 0.75-0.84) (Figure 1, dashed line). Survival analysis for low vs. high predicted risk indicated ICD non-benefit rates of 6.0% versus 30.3% at 3-year follow-up, respectively (Figure 2). Conclusion A ML model that combined clinical with ECG features better predicted ICD non-benefit at 3-years in a primary prevention population than currently available risk scores such as the MADIT-ICD score. This approach may provide new tools to support personalized decision making for ICD therapy. Prospective validation is needed to assess the real-world clinical performance of the model.

Objective: To investigate whether the plasma levels of four biomarkers can help predict systolic dysfunction of either ventricle and left ventricular diastolic dysfunction in patients suffering severe aortic stenosis and scheduled for aortic valve replacement. Background: There has recently been an increasing interest for the use of biomarkers in the diagnosis, prognostication, and follow-up of heart diseases. Apart from the N-terminal pro B-type natriuretic peptide (NT-proBNP), the relationships between these biomarkers and the different aspects of heart function remain poorly investigated in patients eligible for cardiac surgery. It is plausible that some biomarkers more specifically reflect the dysfunction of one or the other ventricle, or diastolic dysfunction. Methods: This unspecified and therefore exploratory analysis of a previously published prospective observational trial adheres to the applicable EQUATOR guidelines. After Ethics Committee approval and written informed consent, the plasma levels of NT-proBNP, soluble isoform of suppression of tumorigenicity 2 (sST2), Galectin 3, and growth differentiation factor (GDF-15) were measured in an arterial blood sample drawn before induction of anesthesia in 179 patients undergoing aortic valve replacement for aortic stenosis. The monotonic interactions between the biomarkers and various echocardiographic measurements performed during the pre-cardiopulmonary bypass transesophageal echo exam were investigated using the Spearmann correlation coefficient. Whenever a significant correlation was observed, the ability of the biomarker to predict abnormal heart function was investigated using the area under the receiver operating curve (AUROC). Results: Significant correlations were observed between the level of NT-proBNP and the ejection fraction of the left ventricle (LVEF) (ρ=-0.31, P<0.001), or its global longitudinal strain (GLS) (ρ=-0.40, P<0.001). The NT-proBNP was also correlated with the left atrial (LA) volume (ρ=0.23, P=0.03) and the LA reservoir strain (ρ=-0.33, P<0.001). The correlations between the e/e’ ratio and both the NT-proBNP (ρ=0.38, P<0.001) and the GDF-15 (ρ=0.24, P=0.006) were significant. No correlation was observed between Galectin 3 and sST2 and any of the echo parameters. The ability of NT-proBNP to predict an LVEF < 50% (AUROC=0.66) and a GLS > -20% (AUROC=0.63) was weak. The ability of NT-proBNP to predict an e/e’ ratio > 14 was moderate (AUROC=0.71) and not significantly improved by the addition of GDF-15 (AUROC=0.69, P=0.44) Conclusions: The NT-proBNP had a weak to moderate ability to predict both systolic and diastolic dysfunction of the left ventricle in our patient population. Apart from a weak relationship between GDF-15 and diastolic dysfunction, no significant relationship was observed between any of the other markers tested and the echocardiographic measurements in our patient population.

Introduction: Depressed left ventricular function is associated with mortality after pediatric cardiac arrest (CA). Echocardiographic global longitudinal strain (GLS) is more sensitive for poor outcomes than ejection fraction (EF) and shortening fraction (SF), with few studies in children. We aimed to identify the frequency of impaired GLS after pediatric CA and its association with hospital mortality. We hypothesized abnormal GLS is associated with mortality. Methods: This is a retrospective single-center cohort study of children < 18 years of age treated in the pediatric intensive care unit after in- and out-of-hospital CA, with echocardiogram performed within 24 hours of admission between 2013-2020. Patients with congenital heart disease, extracorporeal support, or inability to measure GLS were excluded. EF and SF measurements were abstracted and GLS measured post hoc by an investigator blinded to outcome. Abnormal left ventricular GLS was defined as >-18%. Correlation between GLS, EF and SF were calculated with Spearman’s rho. Logistic regression tested association of GLS with mortality. Area under the receiver operator characteristic (AUROC) curve was calculated for discriminative utility of GLS, EF, and SF with mortality. Results: GLS was measured in 124 subjects and abnormal in 49 (40%). Subjects with abnormal GLS were older (median 8 vs 2 years, p< 0.001), had more ventricular tachycardia/fibrillation as initial rhythm (18% vs 4%, p=0.008) and higher maximum troponin in 24 hours (median 2.6 ng/mL vs 0.5, p=0.002). There was no difference between arrest location or CPR duration based on GLS. Subjects with abnormal GLS had lower median EF (43% vs 63%) and SF (24% vs 37%), with strong inverse correlation between GLS and EF (rho -0.76) and SF (rho -0.71), all p< 0.001. Abnormal GLS had higher mortality (57% vs 33%, p=0.009) and GLS was associated with mortality when controlling for age and initial rhythm [aOR 1.17 per 1% increase in GLS (95% CI 1.09-1.26), p< 0.001]. GLS (AUROC 0.69), EF (0.71) and SF (0.71) had similar discrimination for mortality. Conclusions: Post-arrest GLS is a novel complementary metric, strongly correlated with EF and SF and associated with mortality. Future prospective studies should investigate the prognostic utility of GLS alongside SF and EF.

MP07-16 COMPUTER VISION ANALYSIS OF UPPER TRACT UROTHELIAL CARCINOMA TO PREDICT HIGH VS LOW GRADE PATHOLOGY

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

Study DesignRetrospective cohort study.ObjectivesProlonged ICU stay is a driver of higher costs and inferior outcomes in Adult Spinal Deformity (ASD) patients. Machine learning (ML) models have recently been seen as a viable method of predicting pre-operative risk but are often ‘black boxes’ that do not fully explain the decision-making process. This study aims to demonstrate ML can achieve similar or greater predictive power as traditional statistical methods and follows traditional clinical decision-making processes.MethodsFive ML models (Decision Tree, Random Forest, Support Vector Classifier, GradBoost, and a CNN) were trained on data collected from a large urban academic center to predict whether prolonged ICU stay would be required post-operatively. 535 patients who underwent posterior fusion or combined fusion for treatment of ASD were included in each model with a 70-20-10 train-test-validation split. Further analysis was performed using Shapley Additive Explanation (SHAP) values to provide insight into each model’s decision-making process.ResultsThe model’s Area Under the Receiver Operating Curve (AUROC) ranged from 0.67 to 0.83. The Random Forest model achieved the highest score. The model considered length of surgery, complications, and estimated blood loss to be the greatest predictors of prolonged ICU stay based on SHAP values.ConclusionsWe developed a ML model that was able to predict whether prolonged ICU stay was required in ASD patients. Further SHAP analysis demonstrated our model aligned with traditional clinical thinking. Thus, ML models have strong potential to assist with risk stratification and more effective and cost-efficient care.

OBJECTIVES:To analyze the available literature on the performance of artificial intelligence-generated clinical models for the prediction of serious life-threatening events in non-ICU adult patients and evaluate their potential clinical usage.DATA SOURCES:The PubMed database was searched for relevant articles in English literature from January 1, 2000, to January 23, 2022. Search terms, including artificial intelligence, machine learning, deep learning, and deterioration, were both controlled terms and free-text terms.STUDY SELECTION:We performed a systematic search reporting studies that showed performance of artificial intelligence-based models with outcome mortality and clinical deterioration.DATA EXTRACTION:Two review authors independently performed study selection and data extraction. Studies with the same outcome were grouped, namely mortality and various forms of deterioration (including ICU admission, adverse events, and cardiac arrests). Meta-analysis was planned in case sufficient data would be extracted from each study and no considerable heterogeneity between studies was present.DATA SYNTHESIS:In total, 45 articles were included for analysis, in which multiple methods of artificial intelligence were used. Twenty-four articles described models for the prediction of mortality and 21 for clinical deterioration. Due to heterogeneity of study characteristics (patient cohort, outcomes, and prediction models), meta-analysis could not be performed. The main reported measure of performance was the area under the receiver operating characteristic (AUROC) (n = 38), of which 33 (87%) had an AUROC greater than 0.8. The highest reported performance in a model predicting mortality had an AUROC of 0.935 and an area under the precision-recall curve of 0.96.CONCLUSIONS:Currently, a growing number of studies develop and analyzes artificial intelligence-based prediction models to predict critical illness and deterioration. We show that artificial intelligence-based prediction models have an overall good performance in predicting deterioration of patients. However, external validation of existing models and its performance in a clinical setting is highly recommended.

AimsAcute myocardial infarction (AMI) remains a leading global cause of mortality, where timely diagnosis is critical to enable early intervention. The 12-lead electrocardiogram (ECG) is a critical tool for AMI detection. While deep learning (DL) models show promise for automated ECG analysis, most prior studies rely on small, curated datasets with limited external validation, limiting their clinical applicability.Methods and resultsWe developed a multimodal DL model (Conv-BiLSTM-Attn) integrating convolutional and recurrent neural networks with an attention mechanism. Using a large, multi-centre dataset of 145 656 ECGs from 96 813 patients across three Swedish hospitals. We trained the model to detect AMI using raw 12-lead ECG signals and demographic inputs (age, sex). Model performance was evaluated under two external validation protocols: generalization across hospitals (GAH) and leave-one-hospital-out (LOHO). The model achieved an area under the receiver operating characteristic (AUROC) of 0.848 (95% CI: 0.84–0.86) and an area under the precision-recall (AUPRC) of 0.456, reflecting class imbalance (∼6–10% AMI prevalence) under the GAH protocol. Subgroup AUROCs ranged from 0.79 to 0.92 across age and sex groups. At the Youden-optimized threshold (0.439), the model showed sensitivity of 0.736, specificity of 0.793, negative predictive value of 0.976, and weighted F1-score of 0.837. Under the LOHO protocol, AUROCs ranged from 0.801–0.849. At Youden thresholds (0.34–0.50), sensitivity ranged from 0.671 to 0.776 and specificity from 0.651 to 0.801, confirming generalizability across sites. Conv-BiLSTM-Attn outperformed benchmark models, with Score-CAM highlighting relevant ST–T segments.ConclusionThis DL model can support accurate and generalizable AMI detection from routine ECGs, with the Conv-BiLSTM-Attn architecture outperforming current benchmark approaches.

Accurate and early prediction of arrhythmias using Electrocardiograms (ECG) presents significant challenges due to the non-stationary nature of ECG signals and inter-patient variability, posing difficulties even for seasoned cardiologists. Deep Learning (DL) methods offer precision in identifying diagnostic ECG patterns for arrhythmias, yet they often lack the transparency needed for clinical application, thus hindering their broader adoption in healthcare. This study introduces an explainable DL-based prediction model using ECG signals to classify nine distinct arrhythmia categories. We evaluated various DL architectures, including ResNet, DenseNet, and VGG16, using raw ECG data. The ResNet34 model emerged as the most effective, achieving an Area Under the Receiver Operating Characteristic (AUROC) of 0.98 and an F1-score of 0.826. Additionally, we explored a hybrid approach that combines raw ECG signals with Heart Rate Variability (HRV) features. Our explainability analysis, utilizing the SHAP technique, identifies the most influential ECG leads for each arrhythmia type and pinpoints critical signal segments for individual disease prediction. This study emphasizes the importance of explainability in arrhythmia prediction models, a critical aspect often overlooked in current research, and highlights its potential to enhance model acceptance and utility in clinical settings.

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

Immunotherapy has improved outcomes for advanced-stage melanoma, however, predictive biomarkers remain limited. We evaluated whether computed tomography (CT) features from multiple anatomical regions could predict immunotherapy response. This study included 157 advanced cutaneous melanoma patients (mean age: 63.3 years; 65.6% male) treated with PD-1 immune checkpoint inhibitor (ICI) singly or in combination with LAG-3 or CTLA-4 ICIs. The primary outcome was 1-year progression-free survival (PFS ≥12 months). Available artificial intelligence (AI) algorithms were applied to pretreatment CT scans to extract and quantify three-dimensional (3D) body composition and thoracic features across abdominal, chest, pelvic regions, and spinal vertebrae. Feature relationship to PFS was assessed. Machine learning (ML) models were used to predict PFS, utilizing only the most important five features to mitigate overfitting. Prediction performance was evaluated using the area under the receiver operating curve (AUROC) with stratified 10-fold cross-validation. Multiple CT features are significantly associated with immunotherapy response. Body tissues at the L5 spinal vertebrae emerged as key predictors. A random forest classifier (RFC) trained on three CT features (L5 bone volume, L5 subcutaneous adipose tissue volume, pelvis visceral adipose tissue density) and two clinical variables achieved a mean AUROC of 0.83 (95% CI: 0.72-0.94). A logistic regression model using the same features yielded an AUROC of 0.82 (95% CI: 0.74-0.91), significantly outperforming a clinical-only model (AUROC 0.65; 95% CI: 0.56-0.74; p=0.006). Integrating imaging features from multiple anatomical regions can improve the prediction of immunotherapy response in advanced melanoma.

Background: Substance misuse is a heterogeneous and complex set of behavioral conditions that are highly prevalent in hospital settings and frequently co-occur. Few solutions exist to comprehensively and reliably identify these conditions hospital-wide to prioritize care and guide treatment. The aim is apply natural language processing (NLP) to admission notes in the electronic health record (EHR) to accurately screen for substance misuse. Methods: The reference dataset was derived from a hospital-wide program that used structured diagnostic interviews to manually screen admitted patients over 26 months (n=54,915). Temporal validation was provided over the subsequent 12 months (n=16,917) and external validation at a separate health system (n=1,991). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 hours of notes in the EHR were mapped to standardized medical vocabulary and fed into neural network models. The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating curve (AUROC). Findings: The model was trained on a cohort that had 3.5% (n=1,921) with any type of substance misuse. Nearly 11% of patients with substance misuse had more than one type of misuse. The multi-label convolutional neural network classifier had an average AUROC of 0.97 (95% CI: 0.96, 0.98) during temporal validation for all types of substance misuse. The model was well calibrated and demonstrated good face validity with model features containing explicit mentions of aberrant drug-taking behavior. The false-negative and false-positive rates were similar between non-Hispanic Black and non-Hispanic White groups. In external validation, the AUROC for alcohol and opioid misuse remained above 0.85. Interpretation: We developed a novel and accurate approach to leveraging the first 24 hours of EHR notes for screening multiple types of substance misuse. Funding Information: Research reported in this publication was supported by the National Institute On Drug Abuse of the National Institutes of Health under Award Numbers R01-DA051464 (MA), K23-AA024503 (MA), UL1-TR002389 (NK), KL2-TR002387 (NK), R01-DA041071 (NK), UG1-DA049467 (NK), R01-LM010090 (DD), R01-LM012973 (DD), K12-HS-026385 (HT), and R01-GM123193 (MMC). Declaration of Interests: Dr. Churpek has a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients, and has received research support from EarlySense (Tel Aviv, Israel). All other authors have nothing to declare. Ethics Approval Statement: This study was approved by the Institutional Review Board at RUMC and LUMC.

Evaluating artificial intelligence (AI) models in clinical medicine requires more than conventional metrics such as accuracy, Area Under the Receiver Operating Characteristic (AUROC), or F1-score, which often overlook key considerations such as fairness, reliability, and real-world utility. We introduce CUES as a multiplicative composite score for clinical prediction models; it is defined as CUES=(C⋅U⋅E⋅S)1/4, where C represents calibration, U integrated clinical utility, E equity across patient subpopulations, and S sampling stability. We formally establish boundedness, monotonicity, and differentiability on the domain (0,1]4, derive first-order sensitivity relations, and provide asymptotic approximations for its sampling distribution via the delta method. To facilitate inference, we propose bootstrap procedures for constructing confidence intervals and for comparative model evaluation. Analytic examples illustrate how CUES can diverge from traditional metrics, capturing dimensions of predictive performance that are essential for clinical reliability but often missed by AUROC or F1-score alone. By integrating multiple facets of clinical utility and robustness, CUES provides a comprehensive tool for model evaluation, comparison, and selection in real-world medical applications.