PATCH

We developed a multimodal ultrasound (US) deep learning (DL) fusion model to automatically classify early fibrosis in patients with chronic kidney disease (CKD). This prospective study included patients with CKD who underwent continuous gray-scale US, superb microvascular imaging, and strain elastography from May to November 2022. According to the pathological tubular atrophy and interstitial fibrosis score, patients were divided into minimal and mild groups (affected area ≤10% and 11 − 25% of the total cortical volume, respectively). The dataset was divided into training (70%) and test (30%) sets. A DL model combining the features of the three US modes was developed to predict early fibrosis in patients with CKD. We compared these findings with the area under the receiver operating characteristic curve (AUC) of the clinical model by analyzing the receiver operating characteristic curve in the test set. The AUC of single-mode DL based on gray-scale US, superb microvascular imaging, and strain elastography was 0.682, 0.745, and 0.648, respectively, while that of the multimodal US DL model was 0.86. The accuracy, specificity, and sensitivity of the multimodal US DL model were 0.779, 0.767, and 0.796, respectively, and the negative and positive predictive values were 0.842 and 0.706, respectively. The AUC of the multimodal US DL model was significantly better than that of the single-mode DL and clinical models. The DL algorithm developed using multimodal US images can effectively predict early fibrosis in patients with CKD with significantly greater accuracy than single-mode DL or clinical models.

Background Clinicians leverage information from multiple data sources, such as the electrocardiogram (ECG) and echocardiogram to diagnose and manage cardiovascular disease. Deep learning models have shown immense potential to enhance these clinical processes by capturing digital biomarkers of cardiovascular disease from diagnostic modalities. However, the development of multimodal algorithms capable of making joint inferences from multiple modalities, mirroring the clinicians’ decision-making, has remained an unmet need. Purpose We aimed to develop a multimodal foundation model of ECG and echocardiogram as an integrative deep learning approach to evaluating cardiovascular conditions. Methods Twelve-lead ECGs and transthoracic echocardiograms performed at a large academic health system between 2001-2021 were collected. Individuals with paired ECGs and echocardiograms recorded within 3 years were identified and the data was split at the individual level into training, validation, and test sets (80/10/10%). Using paired ECGs and echocardiograms, we developed a deep learning model in a self-supervised setting with contrastive loss and merged embeddings in a joint latent space (Figure 1). We evaluated the performance of this foundation model after fine-tuning for left ventricular systolic dysfunction (LVSD defined as left ventricular ejection fraction of 40% or less), atrial fibrillation, age, and sex as auxiliary tasks. The performance of the multimodal model was compared with those of supervised unimodal ECG-only and echocardiography-only models for each task. Results In total, 4,391,610 videos from 129,424 echocardiography studies paired with 1,094,656 ECGs from 53,876 unique individuals were included (67 ± 14 years of age, 43% female, Figure 1). The multimodal algorithm demonstrated excellent discrimination for disease classification in the test set. In classification of LVSD, for example, it achieved an area under the receiver operating characteristic (AUROC) curve of 0.94, area under precision recall curve (AUPRC) of 0.99 among individuals with LV ejection fraction greater than 40%, and AUPRC of 0.72 among those with LV ejection fraction of 40% or less. The multimodal model consistently outperformed the unimodal algorithms, including the ECG-only and echocardiography-only models in classification of LVSD (AUROC ECG+Echo 0.94, ECG 0.89, Echo 0.89), atrial fibrillation (AUROC ECG+Echo 0.95, ECG, 0.94, Echo 0.75), sex (AUROC ECG+Echo 0.93, ECG 0.87, Echo 0.78) and age (R for variance ECG+Echo 0.75, ECG 0.67, Echo 0.58, Figure 2). Conclusion We have developed a multimodal foundation model for joint inference from ECGs and echocardiograms for diagnosis of cardiovascular conditions. This novel approach outperforms the conventional unimodal models in disease diagnosis and represents an advance in integrating the clinical applications of artificial intelligence.Figure 1 Figure 2

MMCL-CPI: A multi-modal compound-protein interaction prediction model incorporating contrastive learning pre-training

<p dir="ltr">Objective: We aim to develop a multimodal model to predict chronic kidney disease (CKD) in patients with type 2 diabetes mellitus (T2DM), given the limited research on this integrative approach.</p><p dir="ltr">Research Design and Methods: We obtained multimodal datasets from Kyung Hee University Medical Center (South Korea; n=7028; discovery cohort) for training and internal validation and UK Biobank (n=1544; validation cohort) for external validation. CKD was defined based on the international classification of disease-9/10 codes and/or estimated glomerular filtration rate (eGFR) ≤60 mL/min/1.73m². We ensembled various deep learning models and interpreted their predictions using explainable artificial intelligence (AI) methods, including shapley additive explanations (SHAP) and gradient-weighted class activation mapping (Grad-CAM). Subsequently, we investigated the potential association between the model probability and vascular complications.</p><p dir="ltr">Results: The multimodal model, which ensembles visual geometry group 16 (VGG16) and deep neural network (DNN), presented high performance in predicting CKD, with area under the receiver operating characteristic curve (AUROC) of 0.880 (95% CI, 0.806–0.954) in the discovery cohort and 0.722 in the validation cohort. SHAP and Grad-CAM highlighted key predictors, including eGFR and optic disc, respectively. The model probability was associated with an increased risk of macrovascular complications (T1: adjusted hazard ratio, 1.42 [95% CI, 1.06–1.90]; T2: 1.59 [1.17–2.16]; and T3: 1.64 [1.20–2.26]) and microvascular complications (T3: 1.30 [1.02–1.67]).</p><p dir="ltr">Conclusions: Our multimodal AI model integrates fundus images and clinical data from binational cohorts to predict the risk of new-onset CKD within five years and associated vascular complications in patients with T2DM.</p>

Diabetic retinopathy (DR) is one of the leading causes of vision loss in adults and is one of the detrimental side effects of the mass prevalence of Diabetes Mellitus (DM). It is crucial to have an efficient screening method for early diagnosis of DR to prevent vision loss. This paper compares and analyzes the various Machine Learning (ML) techniques, from traditional ML to advanced Deep Learning models. We compared and analyzed the efficacy of Convolutional Neural Networks (CNNs), Capsule Networks (CapsNet), K-Nearest Neighbor (KNN), Support Vector Machine (SVM), decision trees, and Random Forests. This paper also considers determining factors in the evaluation, including contrast enhancements, noise reduction, grayscaling, etc We analyze recent research studies and compare methodologies and metrics, including accuracy, precision, sensitivity, and specificity. The findings highlight the advanced performance of Deep Learning (DL) models, with CapsNet achieving a remarkable accuracy of up to 97.98% and a high precision rate, outperforming other traditional ML methods. The Contrast Limited Adaptive Histogram Equalization (CLAHE) preprocessing technique substantially enhanced the model’s efficiency. Each ML method’s computational requirements are also considered. While most advanced deep learning methods performed better according to the metrics, they are more computationally complex, requiring more resources and data input. We also discussed how datasets like MESSIDOR could be more straightforward and contribute to highly evaluated performance and that there is a lack of consistency regarding benchmark datasets across papers in the field. Using the DL models facilitates accurate early detection for DR screening, can potentially reduce vision loss risks, and improves accessibility and cost-efficiency of eye screening. Further research is recommended to extend our findings by building models with public datasets, experimenting with ensembles of DL and traditional ML models, and considering testing high-performing models like CapsNet.

To compare the diagnostic efficiencies of deep learning single-modal and multi-modal for the classification of benign and malignant breast mass lesions. We retrospectively collected data from 203 patients (207 lesions, 101 benign and 106 malignant) with breast tumors who underwent breast magnetic resonance imaging (MRI) before surgery or biopsy between January 2014 and October 2020. Mass segmentation was performed based on the three dimensions-region of interest (3D-ROI) minimum bounding cube at the edge of the lesion. We established single-modal models based on a convolutional neural network (CNN) including T2WI and non-fs T1WI, the dynamic contrast-enhanced (DCE-MRI) first phase was pre-contrast T1WI (d1), and Phases 2, 4, and 6 were post-contrast T1WI (d2, d4, d6); and Multi-modal fusion models with a Sobel operator (four_mods:T2WI, non-fs-T1WI, d1, d2). Training set (n=145), validation set (n=22), and test set (n=40). Five-fold cross validation was performed. Accuracy, sensitivity, specificity, negative predictive value, positive predictive value, and area under the ROC curve (AUC) were used as evaluation indicators. Delong's test compared the diagnostic performance of the multi-modal and single-modal models. All models showed good performance, and the AUC values were all greater than 0.750. Among the single-modal models, T2WI, non-fs-T1WI, d1, and d2 had specificities of 77.1%, 77.2%, 80.2%, and 78.2%, respectively. d2 had the highest accuracy of 78.5% and showed the best diagnostic performance with an AUC of 0.827. The multi-modal model with the Sobel operator performed better than single-modal models, with an AUC of 0.887, sensitivity of 79.8%, specificity of 86.1%, and positive prediction value of 85.6%. Delong's test showed that the diagnostic performance of the multi-modal fusion models was higher than that of the six single-modal models (T2WI, non-fs-T1WI, d1, d2, d4, d6); the difference was statistically significant (p = 0.043, 0.017, 0.006, 0.017, 0.020, 0.004, all were greater than 0.05). Multi-modal fusion deep learning models with a Sobel operator had excellent diagnostic value in the classification of breast masses, and further increase the efficiency of diagnosis.

Although advancements in remote sensing technology have facilitated quick capture and identification of the source and location of oil spills in water bodies, the presence of other biogenic elements (lookalikes) with similar visual attributes hinder rapid detection and prompt decision making for emergency response. To date, different methods have been applied to distinguish oil spills from lookalikes with limited success. In addition, accurately modeling the trajectory of oil spills remains a challenge. Thus, we aim to provide further insights on the multi-faceted problem by undertaking a holistic review of past and current approaches to marine oil spill disaster reduction as well as explore the potentials of emerging digital trends in minimizing oil spill hazards. The scope of previous reviews is extended by covering the inter-related dimensions of detection, discrimination, and trajectory prediction of oil spills for vulnerability assessment. Findings show that both optical and microwave airborne and satellite remote sensors are used for oil spill monitoring with microwave sensors being more widely used due to their ability to operate under any weather condition. However, the accuracy of both sensors is affected by the presence of biogenic elements, leading to false positive depiction of oil spills. Statistical image segmentation has been widely used to discriminate lookalikes from oil spills with varying levels of accuracy but the emergence of digitalization technologies in the fourth industrial revolution (IR 4.0) is enabling the use of Machine learning (ML) and deep learning (DL) models, which are more promising than the statistical methods. The Support Vector Machine (SVM) and Artificial Neural Network (ANN) are the most used machine learning algorithms for oil spill detection, although the restriction of ML models to feed forward image classification without support for the end-to-end trainable framework limits its accuracy. On the other hand, deep learning models’ strong feature extraction and autonomous learning capability enhance their detection accuracy. Also, mathematical models based on lagrangian method have improved oil spill trajectory prediction with higher real time accuracy than the conventional worst case, average and survey-based approaches. However, these newer models are unable to quantify oil droplets and uncertainty in vulnerability prediction. Considering that there is yet no single best remote sensing technique for unambiguous detection and discrimination of oil spills and lookalikes, it is imperative to advance research in the field in order to improve existing technology and develop specialized sensors for accurate oil spill detection and enhanced classification, leveraging emerging geospatial computer vision initiatives.

BackgroundFine-grained classification deals with data with a large degree of similarity, such as cat or bird species, and similarly, knee osteoarthritis severity classification [Kellgren–Lawrence (KL) grading] is one such fine-grained classification task. Recently, a plug-in module (PIM) that can be integrated into convolutional neural-network-based or transformer-based networks has been shown to provide strong discriminative regions for fine-grained classification, with results that outperformed the previous deep learning models. PIM utilizes each pixel of an image as an independent feature and can subsequently better classify images with minor differences. It was hypothesized that, as a fine-grained classification task, knee osteoarthritis severity may be classified well using PIMs. The aim of the study was to develop this automated knee osteoarthritis classification model.MethodsA deep learning model that classifies knee osteoarthritis severity of a radiograph was developed utilizing PIMs. A retrospective analysis on prospectively collected data was performed. The model was trained and developed using the Osteoarthritis Initiative dataset and was subsequently tested on an independent dataset, the Multicenter Osteoarthritis Study (test set size: 17,040). The final deep learning model was designed through an ensemble of four different PIMs.ResultsThe accuracy was the lowest for KL grade 1 (43%) and the highest for KL grade 4 (96%).ConclusionsThe ensemble of PIMs could classify knee osteoarthritis severity using simple radiographs with a fine accuracy. Although improvements will be needed in the future, the model has been proven to have the potential to be clinically useful.

To develop a deep learning (DL) model to detect morphologic patterns of diabetic macular edema (DME) based on optical coherence tomography (OCT) images. In the training set, 12,365 OCT images were extracted from a public data set and an ophthalmic center. A total of 656 OCT images were extracted from another ophthalmic center for external validation. The presence or absence of three OCT patterns of DME, including diffused retinal thickening, cystoid macular edema, and serous retinal detachment, was labeled with 1 or 0, respectively. A DL model was trained to detect three OCT patterns of DME. The occlusion test was applied for the visualization of the DL model. Applying 5-fold cross-validation method in internal validation, the area under the receiver operating characteristic curve for the detection of three OCT patterns (i.e., diffused retinal thickening, cystoid macular edema, and serous retinal detachment) was 0.971, 0.974, and 0.994, respectively, with an accuracy of 93.0%, 95.1%, and 98.8%, respectively, a sensitivity of 93.5%, 94.5%, and 96.7%, respectively, and a specificity of 92.3%, 95.6%, and 99.3%, respectively. In external validation, the area under the receiver operating characteristic curve was 0.970, 0.997, and 0.997, respectively, with an accuracy of 90.2%, 95.4%, and 95.9%, respectively, a sensitivity of 80.1%, 93.4%, and 94.9%, respectively, and a specificity of 97.6%, 97.2%, and 96.5%, respectively. The occlusion test showed that the DL model could successfully identify the pathologic regions most critical for detection. Our DL model demonstrated high accuracy and transparency in the detection of OCT patterns of DME. These results emphasized the potential of artificial intelligence in assisting clinical decision-making processes in patients with DME.

In this article, we introduce the concept of model interpretability, review its applications in deep learning models for clinical ophthalmology, and discuss its role in the integration of artificial intelligence in healthcare. The advent of deep learning in medicine has introduced models with remarkable accuracy. However, the inherent complexity of these models undermines its users' ability to understand, debug and ultimately trust them in clinical practice. Novel methods are being increasingly explored to improve models' 'interpretability' and draw clearer associations between their outputs and features in the input dataset. In the field of ophthalmology, interpretability methods have enabled users to make informed adjustments, identify clinically relevant imaging patterns, and predict outcomes in deep learning models. Interpretability methods support the transparency necessary to implement, operate and modify complex deep learning models. These benefits are becoming increasingly demonstrated in models for clinical ophthalmology. As quality standards for deep learning models used in healthcare continue to evolve, interpretability methods may prove influential in their path to regulatory approval and acceptance in clinical practice.

Recent advances in deep learning have shown many successful stories in smart healthcare applications with data-driven insight into improving clinical institutions’ quality of care. Excellent deep learning models are heavily data-driven. The more data trained, the more robust and more generalizable the performance of the deep learning model. However, pooling the medical data into centralized storage to train a robust deep learning model faces privacy, ownership, and strict regulation challenges. Federated learning resolves the previous challenges with a shared global deep learning model using a central aggregator server. At the same time, patient data remain with the local party, maintaining data anonymity and security. In this study, first, we provide a comprehensive, up-to-date review of research employing federated learning in healthcare applications. Second, we evaluate a set of recent challenges from a data-centric perspective in federated learning, such as data partitioning characteristics, data distributions, data protection mechanisms, and benchmark datasets. Finally, we point out several potential challenges and future research directions in healthcare applications.

Three-dimensional multimodal imaging for predicting early recurrence of hepatocellular carcinoma after surgical resection.

The classification of coffee beans is crucial for maintaining quality and consistency within the coffee industry. Manual inspection, however, is labor-intensive, error-prone, and susceptible to human biases. To address these challenges, this study aims to automate coffee bean classification using deep learning models to improve accuracy and efficiency. Four pre-trained models—Xception, ResNet50V2, EfficientNetB0, and VGG16—were evaluated for predicting post-roasting coffee bean quality based on two datasets: a Kaggle dataset and a self-collected dataset with an image scanner. The datasets included images of coffee beans at four roast levels: dark, green, light, and medium. The models were trained and tested using standard deep learning techniques, with performance assessed through metrics such as accuracy, precision, recall, and F1-score. The results demonstrated that Xception and EfficientNetB0 achieved the highest classification performance. On the Kaggle dataset, both models achieved 100% accuracy, while on the self-collected dataset, Xception achieved 99.3%, and EfficientNetB0 achieved 99.07%. These findings underscore the robustness of applying deep learning models in automating coffee quality control, reducing human intervention, and enhancing classification reliability.

Frailty is a geriatric syndrome that may result in poor health outcomes such as hospitalization, disability, psychological distress, and reduced life satisfaction, and it is also associated with higher healthcare costs. The aim of this study is to classify frailty in elders at an early stage (pre-frail) to lower the risk of frailty and, hence, improve the quality of life. The other two classes in the classification task are frail and robust (non-frail). To achieve this, a dataset based on gait was utilized, which was recorded by an Inertial Measurement Unit (IMU) sensor, including gyroscope and accelerometer data. In this study, two approaches were assessed: the first used advanced Deep Learning (DL) algorithms to analyze raw IMU signals, and the second used conventional Machine Learning (ML) methods with hand-crafted features. The DL model, i.e., InceptionTime, beat the other algorithms in the DL approach with a remarkable test accuracy of 98%. On the ML side, Random Forest reported the most successful ML method, which achieved a test accuracy of 63.3%. For a careful assessment of the models, other evaluation metrics like Precision, Recall, and F1-score were also evaluated. The evaluation of both approaches produces research benefits for the classification of frailty in older people and allows for the investigation of new areas, promoting deeper comprehension and well-informed decision-making, particularly in healthcare systems.

Quantization-aware training for low precision photonic neural networks