Systematic evaluation of single-cell multimodal data integration enhances cell type resolution and discovery of clinically relevant states in complex tissues

Multimodal data integration has emerged as a transformative approach in the health care sector, systematically combining complementary biological and clinical data sources such as genomics, medical imaging, electronic health records, and wearable device outputs. This approach provides a multidimensional perspective of patient health that enhances the diagnosis, treatment, and management of various medical conditions. This viewpoint presents an overview of the current state of multimodal integration in health care, spanning clinical applications, current challenges, and future directions. We focus primarily on its applications across different disease domains, particularly in oncology and ophthalmology. Other diseases are briefly discussed due to the few available literature. In oncology, the integration of multimodal data enables more precise tumor characterization and personalized treatment plans. Multimodal fusion demonstrates accurate prediction of anti–human epidermal growth factor receptor 2 therapy response (area under the curve=0.91). In ophthalmology, multimodal integration through the combination of genetic and imaging data facilitates the early diagnosis of retinal diseases. However, substantial challenges remain regarding data standardization, model deployment, and model interpretability. We also highlight the future directions of multimodal integration, including its expanded disease applications, such as neurological and otolaryngological diseases, and the trend toward large-scale multimodal models, which enhance accuracy. Overall, the innovative potential of multimodal integration is expected to further revolutionize the health care industry, providing more comprehensive and personalized solutions for disease management.

Motivation: Advances in single-cell measurement techniques provide rich multimodal data, which helps us to explore the life state of cells more deeply. However, multimodal integration, or, learning joint embeddings from multimodal data remains a current challenge. The difficulty in integrating unpaired single-cell multimodal data is that different modalities have different feature spaces, which easily leads to information loss in joint embedding. And few existing methods have fully exploited and fused the information in single-cell multimodal data. Result: In this study, we propose CoVEL, a deep learning method for unsupervised integration of single-cell multimodal data. CoVEL learns single-cell representations from a comprehensive view, including regulatory relationships between modalities, fine-grained representations of cells, and relationships between different cells. The comprehensive view embedding enables CoVEL to remove the gap between modalities while protecting biological heterogeneity. Experimental results on multiple public datasets show that CoVEL is accurate and robust to single-cell multimodal integration. Data availability: https://github.com/shapsider/scintegration.

Prognosis prediction is crucial for non-small cell lung cancer (NSCLC) treatment planning. While tumor hypoxia significantly impacts patient outcomes, identifying hypoxic genomic markers remains challenging. This study sought to identify hypoxic computed tomography (CT) radiomic features and create an artificial intelligence (AI) model for NSCLC through the integration of multi-modal data. In total, 452 NSCLC patients were enrolled in this study, including patients from The Second Affiliated Hospital of Soochow University (SC, n=112), The Cancer Genome Atlas (TCGA)-NSCLC dataset (n=74), the radiogenomics dataset (n=130), and the Gene Expression Omnibus (GEO) datasets (GSE19188: n=82, and GSE87340: n=54). Hypoxia status was classified using optimized cut-off values of hypoxia enrichment scores, which were calculated through single-sample gene set enrichment analysis (ssGSEA) of hypoxic genes. Radiomic features were extracted using three-dimensional (3D)-Slicer software. The least absolute shrinkage and selection operator (LASSO) algorithm was used to identify hypoxic CT radiomic features. A model named ssuBERT (semantic structured unit embedded in Bidirectional Encoder Representations from Transformers) was developed to analyze electronic health records (EHRs). An AI model for overall survival prediction was constructed by integrating CT radiomic features, ssuBERT features, and clinical data, and evaluated using five-fold cross-validation. Higher hypoxia levels were correlated with worse survival outcomes. Twenty-eight radiomic features showed significant discriminatory power in detecting hypoxia status with an area under the curve (AUC) of 0.8295. The ssuBERT model achieved a weighted accuracy of 0.945 in recognizing semantic structured units in EHRs. The EHR model exhibited superior predictive performance among the single-modal models with an AUC of 0.7662. However, the multi-modal AI model had the highest average AUC of 0.8449 and an F1 score of 0.7557. The AI model demonstrated potential in predicting NSCLC patient prognosis through multi-modal data integration, warranting further validation.

It is crucial to accurately predict the disease progression of systemic arterial hypertension in order to determine the most effective therapeutic strategy. To achieve this, we have employed a multimodal data-integration approach to predict the longitudinal progression of new-onset systemic arterial hypertension patients with suspected obstructive sleep apnea (OSA) at the individual level. We developed and validated a predictive nomogram model that utilizes multimodal data, consisting of clinical features, laboratory tests, and sleep monitoring data. We assessed the probabilities of major adverse cardiac and cerebrovascular events (MACCEs) as scores for participants in longitudinal cohorts who have systemic arterial hypertension and suspected OSA. In this cohort study, MACCEs were considered as a composite of cardiac mortality, acute coronary syndrome and nonfatal stroke. The least absolute shrinkage and selection operator (LASSO) regression and multiple Cox regression analyses were performed to identify independent risk factors for MACCEs among these patients. 448 patients were randomly assigned to the training cohort while 189 were assigned to the verification cohort. Four clinical variables were enrolled in the constructed nomogram: age, diabetes mellitus, triglyceride, and apnea-hypopnea index (AHI). This model accurately predicted 2-year and 3-year MACCEs, achieving an impressive area under the receiver operating characteristic (ROC) curve of 0.885 and 0.784 in the training cohort, respectively. In the verification cohort, the performance of the nomogram model had good discriminatory power, with an area under the ROC curve of 0.847 and 0.729 for 2-year and 3-year MACCEs, respectively. The correlation between predicted and actual observed MACCEs was high, provided by a calibration plot, for training and verification cohorts. Our study yielded risk stratification for systemic arterial hypertension patients with suspected OSA, which can be quantified through the integration of multimodal data, thus highlighting OSA as a spectrum of disease. This prediction nomogram could be instrumental in defining the disease state and long-term clinical outcomes.

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.

Joint fusion of EHR and ECG data using attention-based CNN and ViT for predicting adverse clinical endpoints in percutaneous coronary intervention patients.

The transformative role of single-cell analysis in multifactorial disorders research.

This study integrates multimodal metabolomic data from three platforms—LC–MS, GC–MS, and NMR—to systematically identify biomarkers distinguishing breast cancer subtypes. A feedforward attention-based deep learning model effectively selected 99 significant metabolites, outperforming traditional static methods in classification performance and biomarker consistency. By combining data from diverse platforms, the approach captured a comprehensive metabolic profile while maintaining biological relevance. Self-organizing map analysis revealed distinct metabolic signatures for each subtype, highlighting critical pathways. Group 1 (ER/PR-positive, HER2-negative) exhibited elevated serine, tyrosine, and 2-aminoadipic acid levels, indicating enhanced amino acid metabolism supporting nucleotide synthesis and redox balance. Group 3 (triple-negative breast cancer) displayed increased TCA cycle intermediates, such as α-ketoglutarate and malate, reflecting a metabolic shift toward energy production and biosynthesis to sustain aggressive proliferation. In Group 4 (HER2-enriched), elevated phosphatidylcholines and phosphatidylethanolamines suggested upregulated mono-unsaturated phospholipid biosynthesis. The study provides a framework for leveraging multimodal data integration, attention-based feature selection, and self-organizing map analysis to identify biologically meaningful biomarkers.

The Big Data era in Biomedical research has resulted in large-cohort data repositories such as The Cancer Genome Atlas (TCGA). These repositories routinely contain hundreds of matched patient samples for genomic, proteomic, imaging, and clinical data modalities, enabling holistic and multi-modal integrative analysis of human disease. Using TCGA renal and ovarian cancer data, we conducted a novel investigation of multi-modal data integration by combining histopathological image and RNA-seq data. We compared the performances of two integrative prediction methods: majority vote and stacked generalization. Results indicate that integration of multiple data modalities improves prediction of cancer grade and outcome. Specifically, stacked generalization, a method that integrates multiple data modalities to produce a single prediction result, outperforms both single-data-modality prediction and majority vote. Moreover, stacked generalization reveals the contribution of each data modality (and specific features within each data modality) to the final prediction result and may provide biological insights to explain prediction performance.

To develop and validate a predictive combined model for metastasis in patients with clear cell renal cell carcinoma (ccRCC) by integrating multimodal data. In this retrospective study, the clinical and imaging data (CT and ultrasound) of patients with ccRCC confirmed by pathology from three tertiary hospitals in different regions were collected from January 2013 to January 2023. We developed three models, including a clinical model, a radiomics model, and a combined model. The performance of the model was determined based on its discriminative power and clinical utility. The evaluation indicators included area under the receiver operating characteristic curve (AUC) value, accuracy, sensitivity, specificity, negative predictive value, positive predictive value and decision curve analysis(DCA) curve. A total of 251 patients were evaluated. Patients (n = 166) from Shandong University Qilu Hospital (Jinan) were divided into the training cohort, of which 50 patients developed metastases; patients (n = 37) from Shandong University Qilu Hospital (Qingdao) were used as internal testing, of which 15 patients developed metastases; patients (n = 48) from Changzhou Second People's Hospital were used as external testing, of which 13 patients developed metastases. In the training set, the combined model showed the highest performance (AUC, 0.924) in predicting lymph node metastasis(LNM), while the clinical and radiomics models both had AUCs of 0.845 and 0.870, respectively. In the internal testing, the combined model had the highest performance (AUC, 0.877) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.726 and 0.836, respectively. In the external testing, the combined model had the highest performance (AUC, 0.849) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.708 and 0.804, respectively. The DCA curve showed that the combined model had a significant prediction probability in predicting the risk of LNM in ccRCC patients compared with the clinical model or the radiomics model. The combined model was superior to the clinical and radiomics models in predicting LNM in ccRCC patients.

The core value of AI intelligent analysis in the classroom lies in its ability to achieve intelligent diagnosis and dynamic feedback on the teaching process through a three-dimensional collaborative mechanism of “objectives/features/algorithms”. By leveraging multimodal data collection and analysis, it assists teachers in optimizing instructional design and adjusting teaching strategies, while simultaneously monitoring students' learning states and emotional changes in real-time, providing personalized learning support. However, in practice, AI intelligent analysis faces three major challenges: the lack of standardization in multimodal data integration, the limited interpretability of analysis results, and the ambiguity in the compliance of data collection. To address these issues, it is necessary to optimize the application of AI intelligent analysis in teaching from three perspectives—technology, education, and governance—thereby achieving a deeper transformation from “common adaptability” to “precise empowerment”.

BackgroundDetermination of batch-to-batch consistency of botanical drugs (BDs) has long been the bottleneck in quality evaluation primarily due to the chemical diversity inherent in BDs. This diversity presents an obstacle to achieving comprehensive standardization for BDs. Basically, a single detection mode likely leads to substandard analysis results as different classes of structures always possess distinct physicochemical properties. Whereas representing a workaround for multi-target standardization using multi-modal data, data processing for information from diverse sources is of great importance for the accuracy of classification.MethodsIn this research, multi-modal data of 78 batches of Guhong injections (GHIs) consisting of 52 normal and 26 abnormal samples were acquired by employing HPLC-UV, -ELSD, and quantitative 1H NMR (q1HNMR), of which data obtained was then individually used for Pearson correlation coefficient (PCC) calculation and partial least square-discriminant analysis (PLS-DA). Then, a mid-level data fusion method with data containing qualitative and quantitative information to establish a support vector machine (SVM) model for evaluating the batch-to-batch consistency of GHIs.ResultsThe resulting outcomes showed that datasets from one detection mode (e.g., data from UV detectors only) are inadequate for accurately assessing the product's quality. The mid-level data fusion strategy for the quality evaluation enabled the classification of normal and abnormal batches of GHIs at 100% accuracy.ConclusionsA quality assessment strategy was successfully developed by leveraging a mid-level data fusion method for the batch-to-batch consistency evaluation of GHIs. This study highlights the promising utility of data from different detection modes for the quality evaluation of BDs. It also reminds manufacturers and researchers about the advantages of involving data fusion to handle multi-modal data. Especially when done jointly, this strategy can significantly increase the accuracy of product classification and serve as a capable tool for studies of other BDs.

Integrative analysis of large-scale single-cell data collected from diverse cell populations promises an improved understanding of complex biological systems. While several algorithms have been developed for single-cell RNA-sequencing data integration, many lack the scalability to handle large numbers of datasets and/or millions of cells due to their memory and run time requirements. The few tools that can handle large data do so by reducing the computational burden through strategies such as subsampling of the data or selecting a reference dataset to improve computational efficiency and scalability. Such shortcuts, however, hamper the accuracy of downstream analyses, especially those requiring quantitative gene expression information. We present SCEMENT, a SCalablE and Memory-Efficient iNTegration method, to overcome these limitations. Our new parallel algorithm builds upon and extends the linear regression model previously applied in ComBat to an unsupervised sparse matrix setting to enable accurate integration of diverse and large collections of single-cell RNA-sequencing data. Using tens to hundreds of real single-cell RNA-seq datasets, we show that SCEMENT outperforms ComBat as well as FastIntegration and Scanorama in runtime (upto 214× faster) and memory usage (upto 17.5× less). It not only performs batch correction and integration of millions of cells in under 25 min, but also facilitates the discovery of new rare cell types and more robust reconstruction of gene regulatory networks with full quantitative gene expression information. Source code freely available for download at https://github.com/AluruLab/scement, implemented in C++ and supported on Linux.

Single-cell omics enable the dissection of cellular heterogeneity, yet the high dimensionality, inherent noise, and sparsity present significant challenges. These challenges are amplified for rare cell populations, which are often difficult to annotate reliably but can be central to development and disease. As single-cell assays increasingly capture multiple molecular layers, the integrative analysis of such multimodal data further increases complexity. Here, we propose Hydra, a deep generative framework based on an ensemble of variational autoencoders for effective learning of unimodal and multimodal single-cell omics data. Hydra implements interpretable modules for capturing cell-type-specific molecular signatures. The ensemble of such interpretable modules enables reproducible feature selection and robust cell-type annotation, with particular effectiveness for rare populations. We benchmarked Hydra on a repertoire of 21 datasets, including unimodal and multimodal single-cell omics data. Our results demonstrate that Hydra offers comparable to superior performance to several state-of-the-art methods. Finally, we highlight the utility of Hydra in robustly annotating brain cellular subtypes and preserving disease-relevant signatures using our previously published dataset that profiles Alzheimer's disease.

In recent years, unmanned aerial vehicle (UAV) remote sensing systems have advanced rapidly, enabling the effective assessment of crop growth through the processing and integration of multimodal data from diverse sensors mounted on UAVs. UAV-derived multimodal data encompass both multi-source remote sensing data and multi-source non-remote sensing data. This study employs Image Guided Filtering Fusion (GFF) to obtain high-resolution multispectral images (HR-MSs) and selects three vegetation indices (VIs) based on correlation analysis and feature reduction in HR-MS for multi-source sensing data. As a supplement to remote sensing data, multi-source non-remote sensing data incorporate two meteorological conditions: temperature and precipitation. This research aims to establish remote sensing quantitative monitoring models for four crucial growth-physiological indicators during rapeseed (Brassica napus L.) seedling stages, namely, leaf area index (LAI), above ground biomass (AGB), leaf nitrogen content (LNC), and chlorophyll content (SPAD). To validate the monitoring effectiveness of multimodal data, the study constructs four model frameworks based on multimodal data input and employs Support Vector Regression (SVR), Partial Least Squares (PLS), Backpropagation Neural Network (BPNN), and Nonlinear Model Regression (NMR) machine learning models to create winter rapeseed quantitative monitoring models. The findings reveal that the model framework, which integrates multi-source remote sensing data and non-remote sensing data, exhibits the highest average precision (R2 = 0.7454), which is 28%, 14.6%, and 3.7% higher than that of the other three model frameworks, enhancing the model’s robustness by incorporating meteorological data. Furthermore, SVR consistently performs well across various multimodal model frameworks, effectively evaluating the vigor of rapeseed seedlings and providing a valuable reference for rapid, non-destructive monitoring of winter rapeseed.