F-CPI: A Multimodal Deep Learning Approach for Predicting Compound Bioactivity Changes Induced by Fluorine Substitution.

Performance of deep learning in mapping water quality of Lake Simcoe with long-term Landsat archive

Intrusion detection plays a critical role in safeguarding computer networks from malicious activities and unauthorized access. Traditional machine learning (ML) models have been widely employed for intrusion detection, but their effectiveness can be limited due to the complex and evolving nature of network attacks. An efficient Convolutional Neural Network (CNN) that utilizes one-dimensional convolutions, batch normalization, and max pooling layers is designed to capture the attack patterns in network traffic data. By leveraging the inherent feature extraction capabilities of CNNs, the proposed architecture demonstrates improved performance in identifying network intrusions compared to traditional ML models. The effectiveness of the novel CNN architecture is evaluated by conducted comprehensive experiments on two benchmark intrusion detection datasets. The designed CNN performance was compared with four other ML models such as logistic regression, k-nearest neighbors, decision trees, and random forest. The experimental results revealed that the proposed CNN architecture outperformed the traditional ML models with highest accuracy of 97.68% and 97.39% and highest F1-score of 97.68% and 97.40% for. UNSW-NB15 and CICIDS2017 datasets, respectively. The proposed CNN architecture serves as a promising alternative to traditional ML models, offering improved performance in identifying and mitigating network attacks.

This paper provides a comprehensive review of multimodal deep learning models that utilize conversational data to detect mental health disorders. In addition to discussing models based on the Transformer, such as BERT (Bidirectional Encoder Representations from Transformers), this paper addresses models that existed prior to the Transformer, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The paper covers the application of these models in the construction of multimodal deep learning systems to detect mental disorders. In addition, the difficulties encountered by multimodal deep learning systems are brought up. Furthermore, the paper proposes research directions for enhancing the performance and robustness of these models in mental health applications. By shedding light on the potential of multimodal deep learning in mental health care, this paper aims to foster further research and development in this critical domain.

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

Background<p>Clear-cell renal cell carcinoma (ccRCC) is common and associated with substantial mortality. TNM stage and histopathological grading have been the sole determinants of a patient’s prognosis for decades and there are few prognostic biomarkers used in clinical routine. Management of ccRCC involves multiple disciplines such as urology, radiology, oncology, and pathology and each of these specialties generates highly complex medical data. Here, artificial intelligence (AI) could prove extremely powerful to extract meaningful information to benefit patients.</p>Objective<p>In the study, we developed and evaluated a multimodal deep learning model (MMDLM) for prognosis prediction in ccRCC.</p>Design, Setting, and Participants<p>Two mixed cohorts of non-metastatic and metastatic ccRCC patients were used: (1) The Cancer Genome Atlas cohort including 230 patients and (2) the Mainz cohort including 18 patients with ccRCC. For each of these patients, we trained the MMDLM on multiscale histopathological images, CT/MRI scans, and genomic data from whole exome sequencing.</p>Outcome Measurements and Statistical Analysis<p>Outcome measurements included Harrell’s concordance index (C-index) and also various performance parameters for predicting the 5-year survival status (5YSS). Different visualization techniques were used to make our model more transparent.</p>Results<p>The MMDLM showed great performance in predicting the prognosis of ccRCC patients with a mean C-index of 0.7791 and a mean accuracy of 83.43%. Training on a combination of data from different sources yielded significantly better results compared to when only one source was used. Furthermore, the MMDLM’s prediction was an independent prognostic factor outperforming other clinical parameters.</p>Interpretation<p>Multimodal deep learning can contribute to prognosis prediction in ccRCC and potentially help to improve the clinical management of this disease.</p>Patient Summary<p>An AI-based computer program can analyze various medical data (microscopic images, CT/MRI scans, and genomic data) simultaneously and thereby predict the survival time of patients with renal cancer.</p>

To develop and validate a multimodal deep learning model that predicts treatment responses to intravitreal anti-vascular endothelial growth factor (anti-VEGF) injections in patients with diabetic macular oedema (DMO) by combining optical coherence tomography images and clinical data. This study included 107 DMO patients who received three consecutive anti-VEGF treatments. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity. The model's predictions were compared with those of retinal specialists. Among 107 patients, 65 showed good response and 42 showed poor response to treatment. The multimodal model achieved an AUROC of 0.962 (95% CI, 0.945-0.979), accuracy of 0.953 (95% CI, 0.933-0.973), sensitivity of 0.969 (95% CI, 0.951-0.987), and specificity of 0.928 (95% CI, 0.903-0.953) in the internal validation. The model outperformed retinal specialists, who achieved accuracies ranging from 0.571 to 0.857. The multimodal deep learning model demonstrated high accuracy in predicting anti-VEGF treatment responses in DMO patients. This approach could enable more personalised treatment strategies and optimal resource utilisation in ophthalmological care. Further validation with larger, multicentre datasets is warranted to confirm its clinical utility.

Unveiling pedestrian injury risk factors through integration of urban contexts using multimodal deep learning

Accurate rainfall forecasting is crucial in activities such as agricultural planning, water resource management, and flood preparedness, so failure to accurately forecast rainfall might undermine these activities. This study aimed to improve the accuracy of daily rainfall prediction in Hai district, Kilimanjaro region, Tanzania, a region characterised by a bimodal rainfall pattern, by comparing rainfall prediction accuracies of Traditional Machine Learning (TML) and Deep Learning (DL) models. The 25-year daily rainfall dataset for Hai district was first pre-processed by cleaning, normalisation, and splitting into training (first 60%), validation (next 20%), and test (last 20%) sets. TML models (K-Nearest Neighbours (KNN) through its variant KNN Regressor and Support Vector Machine (SVM) through its variant Support Vector Regressor (SVR)) were developed, trained, and their performances compared with developed and trained DL models (Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU)). Each model’s test performance was evaluated using the Root Mean Squared Error (RMSE) metric. The findings revealed that the DL models, LSTM and GRU, achieved test RMSE scores of 6.35 mm and 6.49 mm, respectively, while the TML models, KNN and SVM, achieved test RMSE scores of 6.61 mm and 10.42 mm, respectively. The study demonstrated that DL models are more effective for accurate rainfall forecasting than TML models in regions with complex rainfall patterns. The insights gained will guide the development of more reliable, location-specific weather prediction systems, ultimately supporting informed decision-making in agriculture, environmental management, and flood risk reduction in Tanzania and other regions facing similar climatic challenges

BackgroundAccurate prediction of pathological complete response (pCR) to neoadjuvant chemotherapy has significant clinical utility in the management of breast cancer treatment. Although multimodal deep learning models have shown promise for predicting pCR from medical imaging and other clinical data, their adoption has been limited due to challenges with interpretability and generalizability across institutions.MethodsWe developed a multimodal deep learning model combining post contrast-enhanced whole-breast MRI at pre- and post-treatment timepoints with non-imaging clinical features. The model integrates 3D convolutional neural networks and self-attention to capture spatial and cross-modal interactions. We utilized two public multi-institutional datasets to perform internal and external validation of the model. For model training and validation, we used data from the I-SPY 2 trial (N = 660). For external validation, we used the I-SPY 1 dataset (N = 114).ResultsOf the 660 patients in I-SPY 2, 217 patients achieved pCR (32.88%). Of the 114 patients in I-SPY 1, 29 achieved pCR (25.44%). The attention-based multimodal model yielded the best predictive performance with an AUC of 0.73 ± 0.04 on the internal data and an AUC of 0.71 ± 0.02 on the external dataset. The MRI-only model (internal AUC = 0.68 ± 0.03, external AUC = 0.70 ± 0.04) and the non-MRI clinical features-only model (internal AUC = 0.66 ± 0.08, external AUC = 0.71 ± 0.03) trailed in performance, indicating the combination of both modalities is most effective.ConclusionWe present a robust and interpretable deep learning framework for pCR prediction in breast cancer patients undergoing NAC. By combining imaging and clinical data with attention-based fusion, the model achieves strong predictive performance and generalizes across institutions.

Multimodal deep learning model for predicting microsatellite instability in colorectal cancer by contrast-enhanced computed tomography and histopathology.

Early detection and treatment of sarcopenia are crucial for improving patient outcomes, yet current diagnostic methods often lack the accuracy, accessibility, and efficiency needed for widespread clinical use. The aim of this study was to develop an accurate, secure, and evidence-based multimodal AI model using a point-of-care ultrasound (POCUS) framework combining muscle imaging properties with physical performance for sarcopenia diagnosis. The model uses clinical data and POCUS images. Clinical data consisted of age, gender, height, weight, body mass index (BMI) and data on physical performance by Short Physical Performance Battery (SPPB) scores. SPPB scores were chosen since it is recommended by both the European Working Group of Sarcopenia in Older People 2 and the Asian Working Group for Sarcopenia. POCUS data consisted of images from the dominant thigh, focusing on the rectus femoris muscle, using longitudinal and transverse projections. Various Machine Learning (ML) and Deep Learning (DL) algorithms and multimodal architectures were tested. Explainable AI (XAI) methods, including Grad-CAM for ultrasound images and feature-attribution analysis for clinical variables, were integrated to provide transparent interpretation of the multimodal model’s diagnostic decisions. The final model was implemented as part of the Sarcopenia Artificial Intelligence Diagnostic Decision Support System (SAID DSS). Participants (24) were mostly women (63%) with a mean age of 81 years (± 5.2), (age range: 71–91 years) a mean body mass index of 26 kg/m2 (± 4.1), and mean SPPB scores of 5 (± 1.6) and 9 (± 1.6) for sarcopenic and controls. 1060 and 2414 longitudinal and transverse ultrasound events for sarcopenic and control participants, respectively, were used, demonstrating a robust dataset despite the small number of participants. Comprehensive experimental results showed that a feature-level fusion technique using a multilayer perceptron network as classifier and Xception architectures for image feature extraction demonstrated the best performance. The final model yielded a diagnostic accuracy of 85%, an F1-score of 0.85 and an area under the curve (AUC) of 0.84, higher than previous models. This study is the first to introduce a clinically oriented, AI-based multimodal model for sarcopenia detection, demonstrating improved performance over existing approaches. In addition, we provided an explanation of the decisions generated by the best-performing detection model. By integrating this model into SAID DSS, we provide a practical and scalable tool with potential for direct application in clinical workflows, supporting early and accurate identification of sarcopenia. Not applicable.

Objective:Poor outcomes in acute respiratory distress syndrome (ARDS) can be alleviated with tools that support early diagnosis. Current machine learning methods for detecting ARDS do not take full advantage of the multimodality of ARDS pathophysiology. We developed a multimodal deep learning model that uses imaging data, continuously collected ventilation data, and tabular data derived from a patient’s electronic health record (EHR) to make ARDS predictions.Materials and Methods:A chest radiograph (x-ray), at least two hours of ventilator waveform (VWD) data within the first 24 hours of intubation, and EHR-derived tabular data were used from 220 patients admitted to the ICU to train a deep learning model. The model uses pretrained encoders for the x-rays and ventilation data and trains a feature extractor on tabular data. Encoded features for a patient are combined to make a single ARDS prediction. Ablation studies for each modality assessed their effect on the model’s predictive capability.Results:The trimodal model achieved an area under the receiver operator curve (AUROC) of 0.86 with a 95% confidence interval of 0.01. This was a statistically significant improvement (p<0.05) over single modality models and bimodal models trained on VWD+tabular and VWD+x-ray data.Discussion and Conclusion:Our results demonstrate the potential utility of using deep learning to address complex conditions with heterogeneous data. More work is needed to determine the additive effect of modalities on ARDS detection. Our framework can serve as a blueprint for building performant multimodal deep learning models for conditions with small, heterogeneous datasets.

Graph theory provides a robust framework for modelling complex relationships in medical data, enhancing classification accuracy through relational learning. Unlike traditional machine learning (ML) models that treat data points independently, graph-based approaches support structural dependencies to improve feature representation. This study explores the application of Graph Attention Networks (GAT), GraphSAGE, Graph Convolutional Network(GCN) and Graph Isomorphism Network (GIN) for breast cancer classification, employing k-nearest neighbor (KNN) graphs to construct a structured dataset where nodes represent patients and edges capture feature similarities. The effectiveness of the graph-based approaches is evaluated against traditional ML classifiers, including Decision Trees, Random Forest, LightGBM, and XGBoost. Experimental results indicate that GCN, GIN, GAT and GraphSAGE are beat conventional methods, with GCN, GAT and GraphSAGE achieved 100% test-accuracy and with GIN achieved 99.42%, to confirm the percentage of accuracy, authors conducted extensive experiments, including robustness testing by reducing KNN connections, introducing noise, and shuffling train-test splits. Results demonstrate that graph-based models are significantly best than traditional ML models, and these graph-based models maintain same classification accuracy while maintaining stability under robustness tests. The findings confirm that graph-based learning provides a scalable, interpretable, and highly accurate alternative for medical classification tasks, proving its effectiveness in distinguishing between benign and malignant tumors.

Metagenomic sequencing methods provide considerable genomic information regarding human microbiomes, enabling us to discover and understand microbial diseases. Compositional differences have been reported between patients and healthy people, which could be used in the diagnosis of patients. Despite significant progress in this regard, the accuracy of these tools needs to be improved for applications in diagnostics and therapeutics. MDL4Microbiome, the method developed herein, demonstrated high accuracy in predicting disease status by using various features from metagenome sequences and a multimodal deep learning model. We propose combining three different features, i.e., conventional taxonomic profiles, genome-level relative abundance, and metabolic functional characteristics, to enhance classification accuracy. This deep learning model enabled the construction of a classifier that combines these various modalities encoded in the human microbiome. We achieved accuracies of 0.98, 0.76, 0.84, and 0.97 for predicting patients with inflammatory bowel disease, type 2 diabetes, liver cirrhosis, and colorectal cancer, respectively; these are comparable or higher than classical machine learning methods. A deeper analysis was also performed on the resulting sets of selected features to understand the contribution of their different characteristics. MDL4Microbiome is a classifier with higher or comparable accuracy compared with other machine learning methods, which offers perspectives on feature generation with metagenome sequences in deep learning models and their advantages in the classification of host disease status.

Physics-supervised ensemble learning model for predicting failure modes of reinforced concrete columns