Comprehensive View Embedding Learning for Single-Cell Multimodal Integration

Multimodal data integration has emerged as a transformative approach in the health care sector, systematically combining complementary biological and clinical data sources such as genomics, medical imaging, electronic health records, and wearable device outputs. This approach provides a multidimensional perspective of patient health that enhances the diagnosis, treatment, and management of various medical conditions. This viewpoint presents an overview of the current state of multimodal integration in health care, spanning clinical applications, current challenges, and future directions. We focus primarily on its applications across different disease domains, particularly in oncology and ophthalmology. Other diseases are briefly discussed due to the few available literature. In oncology, the integration of multimodal data enables more precise tumor characterization and personalized treatment plans. Multimodal fusion demonstrates accurate prediction of anti–human epidermal growth factor receptor 2 therapy response (area under the curve=0.91). In ophthalmology, multimodal integration through the combination of genetic and imaging data facilitates the early diagnosis of retinal diseases. However, substantial challenges remain regarding data standardization, model deployment, and model interpretability. We also highlight the future directions of multimodal integration, including its expanded disease applications, such as neurological and otolaryngological diseases, and the trend toward large-scale multimodal models, which enhance accuracy. Overall, the innovative potential of multimodal integration is expected to further revolutionize the health care industry, providing more comprehensive and personalized solutions for disease management.

BackgroundThe integration of multimodal single-cell data enables comprehensive organ reference atlases, yet its impact remains largely unexplored, particularly in complex tissues. Using the kidney as an emblematic example of a complex organ, we perform a systematic evaluation of multimodal single-cell integration strategies, with heart tissue used for additional methodological validation.ResultsWe generate a benchmarking dataset for the renal cortex by integrating 3' and 5' scRNA-seq with joint snRNA-seq and snATAC-seq, profiling 119,744 high-quality nuclei/cells from 19 donors. To align cell identities and enable consistent comparisons, we develop the interpretable machine learning tool scOMM (single-cell Omics Multimodal Mapping) and systematically assess integration strategies. "Horizontal" integration of scRNA and snRNA-seq improves cell-type identification, while "vertical" integration of snRNA-seq and snATAC-seq has an additive effect, enhancing resolution in homogeneous populations and difficult-to-identify states. Global integration is especially effective in identifying adaptive states and rare cell types, including WFDC2-expressing Thick Ascending Limb and Norn cells, previously undetected in kidney atlases.ConclusionsOur work establishes a robust framework for multimodal reference atlas generation, advancing single-cell analysis and extending its applicability to diverse tissues.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13059-026-04002-4.

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.

Prognosis prediction is crucial for non-small cell lung cancer (NSCLC) treatment planning. While tumor hypoxia significantly impacts patient outcomes, identifying hypoxic genomic markers remains challenging. This study sought to identify hypoxic computed tomography (CT) radiomic features and create an artificial intelligence (AI) model for NSCLC through the integration of multi-modal data. In total, 452 NSCLC patients were enrolled in this study, including patients from The Second Affiliated Hospital of Soochow University (SC, n=112), The Cancer Genome Atlas (TCGA)-NSCLC dataset (n=74), the radiogenomics dataset (n=130), and the Gene Expression Omnibus (GEO) datasets (GSE19188: n=82, and GSE87340: n=54). Hypoxia status was classified using optimized cut-off values of hypoxia enrichment scores, which were calculated through single-sample gene set enrichment analysis (ssGSEA) of hypoxic genes. Radiomic features were extracted using three-dimensional (3D)-Slicer software. The least absolute shrinkage and selection operator (LASSO) algorithm was used to identify hypoxic CT radiomic features. A model named ssuBERT (semantic structured unit embedded in Bidirectional Encoder Representations from Transformers) was developed to analyze electronic health records (EHRs). An AI model for overall survival prediction was constructed by integrating CT radiomic features, ssuBERT features, and clinical data, and evaluated using five-fold cross-validation. Higher hypoxia levels were correlated with worse survival outcomes. Twenty-eight radiomic features showed significant discriminatory power in detecting hypoxia status with an area under the curve (AUC) of 0.8295. The ssuBERT model achieved a weighted accuracy of 0.945 in recognizing semantic structured units in EHRs. The EHR model exhibited superior predictive performance among the single-modal models with an AUC of 0.7662. However, the multi-modal AI model had the highest average AUC of 0.8449 and an F1 score of 0.7557. The AI model demonstrated potential in predicting NSCLC patient prognosis through multi-modal data integration, warranting further validation.

It is crucial to accurately predict the disease progression of systemic arterial hypertension in order to determine the most effective therapeutic strategy. To achieve this, we have employed a multimodal data-integration approach to predict the longitudinal progression of new-onset systemic arterial hypertension patients with suspected obstructive sleep apnea (OSA) at the individual level. We developed and validated a predictive nomogram model that utilizes multimodal data, consisting of clinical features, laboratory tests, and sleep monitoring data. We assessed the probabilities of major adverse cardiac and cerebrovascular events (MACCEs) as scores for participants in longitudinal cohorts who have systemic arterial hypertension and suspected OSA. In this cohort study, MACCEs were considered as a composite of cardiac mortality, acute coronary syndrome and nonfatal stroke. The least absolute shrinkage and selection operator (LASSO) regression and multiple Cox regression analyses were performed to identify independent risk factors for MACCEs among these patients. 448 patients were randomly assigned to the training cohort while 189 were assigned to the verification cohort. Four clinical variables were enrolled in the constructed nomogram: age, diabetes mellitus, triglyceride, and apnea-hypopnea index (AHI). This model accurately predicted 2-year and 3-year MACCEs, achieving an impressive area under the receiver operating characteristic (ROC) curve of 0.885 and 0.784 in the training cohort, respectively. In the verification cohort, the performance of the nomogram model had good discriminatory power, with an area under the ROC curve of 0.847 and 0.729 for 2-year and 3-year MACCEs, respectively. The correlation between predicted and actual observed MACCEs was high, provided by a calibration plot, for training and verification cohorts. Our study yielded risk stratification for systemic arterial hypertension patients with suspected OSA, which can be quantified through the integration of multimodal data, thus highlighting OSA as a spectrum of disease. This prediction nomogram could be instrumental in defining the disease state and long-term clinical outcomes.

BackgroundDetermination of batch-to-batch consistency of botanical drugs (BDs) has long been the bottleneck in quality evaluation primarily due to the chemical diversity inherent in BDs. This diversity presents an obstacle to achieving comprehensive standardization for BDs. Basically, a single detection mode likely leads to substandard analysis results as different classes of structures always possess distinct physicochemical properties. Whereas representing a workaround for multi-target standardization using multi-modal data, data processing for information from diverse sources is of great importance for the accuracy of classification.MethodsIn this research, multi-modal data of 78 batches of Guhong injections (GHIs) consisting of 52 normal and 26 abnormal samples were acquired by employing HPLC-UV, -ELSD, and quantitative 1H NMR (q1HNMR), of which data obtained was then individually used for Pearson correlation coefficient (PCC) calculation and partial least square-discriminant analysis (PLS-DA). Then, a mid-level data fusion method with data containing qualitative and quantitative information to establish a support vector machine (SVM) model for evaluating the batch-to-batch consistency of GHIs.ResultsThe resulting outcomes showed that datasets from one detection mode (e.g., data from UV detectors only) are inadequate for accurately assessing the product's quality. The mid-level data fusion strategy for the quality evaluation enabled the classification of normal and abnormal batches of GHIs at 100% accuracy.ConclusionsA quality assessment strategy was successfully developed by leveraging a mid-level data fusion method for the batch-to-batch consistency evaluation of GHIs. This study highlights the promising utility of data from different detection modes for the quality evaluation of BDs. It also reminds manufacturers and researchers about the advantages of involving data fusion to handle multi-modal data. Especially when done jointly, this strategy can significantly increase the accuracy of product classification and serve as a capable tool for studies of other BDs.

In recent Datascience and AI studies, multimodality has become a key principle for achieving more accurate and dependable results. Managing multimodal data requires specialized platforms and structures to handle diverse data types from single subjects. The established BIDS (Brain Imaging Data Structure) standard faces limitations in supporting multimodal data, as each dataset is assigned to a single study with unique subject identities, preventing integration of multimodal data from the same individual across multiple studies. To address this limitation, this paper introduces FAIR m-BIDS (FAIR Multimodal Brain Imaging Data Structure), extending conventional BIDS by shifting granularity from dataset level to individual data entities. Each brain data file receives an independent GUId-Key (Global Unique Identifier Key), enabling researchers to select and integrate data items from different modalities and studies into customized multimodal datasets. The proposed structure enhances FAIR principles through improved findability, accessibility, interoperability, and reusability. Global identifiers enable tracking anonymized subject data across multiple datasets and modalities, while maintaining compatibility with conventional BIDS standards for advanced AI and neuroscience research applications.

Joint fusion of EHR and ECG data using attention-based CNN and ViT for predicting adverse clinical endpoints in percutaneous coronary intervention patients.

The Big Data era in Biomedical research has resulted in large-cohort data repositories such as The Cancer Genome Atlas (TCGA). These repositories routinely contain hundreds of matched patient samples for genomic, proteomic, imaging, and clinical data modalities, enabling holistic and multi-modal integrative analysis of human disease. Using TCGA renal and ovarian cancer data, we conducted a novel investigation of multi-modal data integration by combining histopathological image and RNA-seq data. We compared the performances of two integrative prediction methods: majority vote and stacked generalization. Results indicate that integration of multiple data modalities improves prediction of cancer grade and outcome. Specifically, stacked generalization, a method that integrates multiple data modalities to produce a single prediction result, outperforms both single-data-modality prediction and majority vote. Moreover, stacked generalization reveals the contribution of each data modality (and specific features within each data modality) to the final prediction result and may provide biological insights to explain prediction performance.

In recent years, unmanned aerial vehicle (UAV) remote sensing systems have advanced rapidly, enabling the effective assessment of crop growth through the processing and integration of multimodal data from diverse sensors mounted on UAVs. UAV-derived multimodal data encompass both multi-source remote sensing data and multi-source non-remote sensing data. This study employs Image Guided Filtering Fusion (GFF) to obtain high-resolution multispectral images (HR-MSs) and selects three vegetation indices (VIs) based on correlation analysis and feature reduction in HR-MS for multi-source sensing data. As a supplement to remote sensing data, multi-source non-remote sensing data incorporate two meteorological conditions: temperature and precipitation. This research aims to establish remote sensing quantitative monitoring models for four crucial growth-physiological indicators during rapeseed (Brassica napus L.) seedling stages, namely, leaf area index (LAI), above ground biomass (AGB), leaf nitrogen content (LNC), and chlorophyll content (SPAD). To validate the monitoring effectiveness of multimodal data, the study constructs four model frameworks based on multimodal data input and employs Support Vector Regression (SVR), Partial Least Squares (PLS), Backpropagation Neural Network (BPNN), and Nonlinear Model Regression (NMR) machine learning models to create winter rapeseed quantitative monitoring models. The findings reveal that the model framework, which integrates multi-source remote sensing data and non-remote sensing data, exhibits the highest average precision (R2 = 0.7454), which is 28%, 14.6%, and 3.7% higher than that of the other three model frameworks, enhancing the model’s robustness by incorporating meteorological data. Furthermore, SVR consistently performs well across various multimodal model frameworks, effectively evaluating the vigor of rapeseed seedlings and providing a valuable reference for rapid, non-destructive monitoring of winter rapeseed.

The integration of multimodal data and artificial intelligence (AI) in precision agriculture has the potential to significantly improve crop monitoring, management, and overall agricultural productivity. Precision agriculture is advancing rapidly by utilizing multimodal data and intelligent system design. These help to improve decision-making and increase farming efficiency. The main challenges are the fusion of data, introducing AI-powered analytics, and human-computer interaction (HCI) in the field of smart agricultural services. To address the challenges, we investigate how to merge various forms of data. Also, we studied innovative applications to use multimodal data and an intelligent system design to improve crop monitoring and management. This research shows a novel aspect to utilize and explains the difficulties of working with a variety of data sources, such as satellite images, sensor data, weather reports, and farmer inputs, to design an intelligent system. In addition, this research examines how advanced data fusion techniques bring these disparate data types together to provide accurate, real-time insights that help optimize farming practices. AI analytics enable intelligent decision-making and timely alerts, resulting in improved agricultural practices using an advanced predictive machine learning (ML) model. This study adds to the expanding body of knowledge on precision agriculture by introducing novel methodology and practical applications, opening the door for more sustainable and intelligent farming systems.

Traffic prediction, as a core task to alleviate urban congestion and optimize the transport system, has limitations in the integration of multimodal data, making it difficult to comprehensively capture the complex spatio-temporal characteristics of the transport system. Although some studies have attempted to introduce multimodal data, they mostly rely on resource-intensive deep neural network architectures, which have difficultly meeting the demands of practical applications. To this end, we propose a traffic prediction framework based on simple machine learning techniques that effectively integrates property features, amenity features, and emotion features (PAE features). Validated with large-scale real datasets, the method demonstrates excellent prediction performance while significantly reducing computational complexity and deployment costs. This study demonstrates the great potential of simple machine learning techniques in multimodal data fusion, provides an efficient and practical solution for traffic prediction, and offers an effective alternative to resource-intensive deep learning methods, opening up new paths for building scalable traffic prediction systems.

Single-cell omics enable the dissection of cellular heterogeneity, yet the high dimensionality, inherent noise, and sparsity present significant challenges. These challenges are amplified for rare cell populations, which are often difficult to annotate reliably but can be central to development and disease. As single-cell assays increasingly capture multiple molecular layers, the integrative analysis of such multimodal data further increases complexity. Here, we propose Hydra, a deep generative framework based on an ensemble of variational autoencoders for effective learning of unimodal and multimodal single-cell omics data. Hydra implements interpretable modules for capturing cell-type-specific molecular signatures. The ensemble of such interpretable modules enables reproducible feature selection and robust cell-type annotation, with particular effectiveness for rare populations. We benchmarked Hydra on a repertoire of 21 datasets, including unimodal and multimodal single-cell omics data. Our results demonstrate that Hydra offers comparable to superior performance to several state-of-the-art methods. Finally, we highlight the utility of Hydra in robustly annotating brain cellular subtypes and preserving disease-relevant signatures using our previously published dataset that profiles Alzheimer's disease.

To develop and validate a predictive combined model for metastasis in patients with clear cell renal cell carcinoma (ccRCC) by integrating multimodal data. In this retrospective study, the clinical and imaging data (CT and ultrasound) of patients with ccRCC confirmed by pathology from three tertiary hospitals in different regions were collected from January 2013 to January 2023. We developed three models, including a clinical model, a radiomics model, and a combined model. The performance of the model was determined based on its discriminative power and clinical utility. The evaluation indicators included area under the receiver operating characteristic curve (AUC) value, accuracy, sensitivity, specificity, negative predictive value, positive predictive value and decision curve analysis(DCA) curve. A total of 251 patients were evaluated. Patients (n = 166) from Shandong University Qilu Hospital (Jinan) were divided into the training cohort, of which 50 patients developed metastases; patients (n = 37) from Shandong University Qilu Hospital (Qingdao) were used as internal testing, of which 15 patients developed metastases; patients (n = 48) from Changzhou Second People's Hospital were used as external testing, of which 13 patients developed metastases. In the training set, the combined model showed the highest performance (AUC, 0.924) in predicting lymph node metastasis(LNM), while the clinical and radiomics models both had AUCs of 0.845 and 0.870, respectively. In the internal testing, the combined model had the highest performance (AUC, 0.877) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.726 and 0.836, respectively. In the external testing, the combined model had the highest performance (AUC, 0.849) for predicting LNM, while the AUCs of the clinical and radiomics models were 0.708 and 0.804, respectively. The DCA curve showed that the combined model had a significant prediction probability in predicting the risk of LNM in ccRCC patients compared with the clinical model or the radiomics model. The combined model was superior to the clinical and radiomics models in predicting LNM in ccRCC patients.

The core value of AI intelligent analysis in the classroom lies in its ability to achieve intelligent diagnosis and dynamic feedback on the teaching process through a three-dimensional collaborative mechanism of “objectives/features/algorithms”. By leveraging multimodal data collection and analysis, it assists teachers in optimizing instructional design and adjusting teaching strategies, while simultaneously monitoring students' learning states and emotional changes in real-time, providing personalized learning support. However, in practice, AI intelligent analysis faces three major challenges: the lack of standardization in multimodal data integration, the limited interpretability of analysis results, and the ambiguity in the compliance of data collection. To address these issues, it is necessary to optimize the application of AI intelligent analysis in teaching from three perspectives—technology, education, and governance—thereby achieving a deeper transformation from “common adaptability” to “precise empowerment”.