Online Learning for Multimodal Data Fusion With Application to Object Recognition

Multi-modal fusion model for Time-Varying medical Data: Addressing Long-Term dependencies and memory challenges in sequence fusion.

Aiming at raw vibration signal of rolling bearing with long time series, a fault diagnosis model based on multimodal data fusion and deep belief network is proposed in this paper. First, multimodal data composed of artificial features and model features can be obtained by time-frequency domain analysis and unsupervised learning based on restricted Boltzmann machine (RBM). Second, canonical correlation analysis method is used to extract the typical feature pairs from the multimodal data to realize the feature-level multimodal data fusion. Third, deep belief network is applied to extract deep feature mapping between typical feature pairs and fault types. After greedy layer-wise pre-training and fine-tuning, it is available to achieve the trained model for fault diagnosis of rolling bearing. Typical rolling bearing datasets are used to testify the effectiveness of the proposed method. It is verified that the robustness and accuracy of the proposed method are superior to common methods.

Language understanding is multimodal. During human communication, messages are conveyed not only by words in textual form, but also through speech patterns, gestures or facial emotions of the speakers. Therefore, it is crucial to fuse information from different modalities to achieve a joint comprehension. With the rapid progress in the deep learning field, neural networks have emerged as the most popular approach for addressing multimodal data fusion [1, 6, 7, 12]. While these models can effectively combine multimodal features by learning from data, they nevertheless lack an explicit exhibition of how different modalities are related to each other, due to the inherent low interpretability of neural networks [2]. In the meantime, Quantum Theory (QT) has given rise to principled approaches for incorporating interactions between textual features into a holistic textual representation [3, 5, 8, 10], where the concepts of superposition andentanglement have been universally exploited to formulate interactions. The advantages of those models in capturing complicated correlations between textual features have been observed. We hereby propose the research on quantum-inspired multimodal data fusion, claiming that the limitation of multimodal data fusion can be tackled by quantum-driven models. In particular, we propose to employ superposition to formulate intra-modal interactions while the interplay between different modalities is expected to be captured by entanglement measures. By doing so, the interactions within multimodal data may be rendered explicitly in a unified quantum formalism, increasing the performance and interpretability for concrete multimodal tasks. It will also expand the application domains of quantum theory to multimodal tasks where only preliminary efforts have been made [11]. We therefore aim at answering the following research question: RQ. Can we fuse multimodal data with quantum-inspired models? To answer this question, we propose to fuse multimodal data with complex-valued neural networks, motivated by the theoretical link between neural networks and quantum theory [4] and advances in complex-valued neural networks [9]. Our model begins with a separate complex-valued embedding learned for each unimodal data based on the existing works [5, 10] which inherently assumes superposition between intra-modal features. Then we construct a many-body system in entangled state for multimodal data, where cross-modality interactions are naturally reflected by entanglement measures. Quantum measurement operators are applied to the entanglement state to address a concrete multimodal task at hand. The whole process is instrumented by a complex-valued neural network, which is able to learn how multimodal features are combined from data, and at the same time explain the combination by means of quantum superposition and entanglement measures. We plan to examine our proposed models on CMU-MOSI [12] and CMU-MOSEI [1] which are benchmarking multimodal sentiment analysis datasets. The dataset targets at classifying sentiment into 2, 5 or 7 classes with the input of textual, visual and acoustic features. We expect to see comparable effectiveness to state-of-the-art models, and we will explore superposition and entanglement measures to better understand the inter-modal interactions.

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

This review paper provides a comprehensive overview of logistics behavior analysis and predictive modeling methods based on multimodal data fusion. The increasing complexity of modern logistics necessitates advanced analytical techniques to optimize operations, enhance efficiency, and improve decision-making. Multimodal data fusion, which integrates diverse data sources such as GPS tracking, sensor data, transaction records, and weather information, offers a powerful approach to understanding and predicting logistics behavior. This paper explores the historical evolution of logistics analytics, focusing on the shift from traditional statistical methods to contemporary machine learning techniques. Core themes examined include data acquisition and preprocessing, feature engineering for logistics applications, and the application of various predictive models such as regression models, time series analysis, and deep learning algorithms. A comparative analysis highlights the strengths and weaknesses of different modeling approaches, addressing challenges related to data heterogeneity, scalability, and real-time processing. Furthermore, the review identifies emerging trends and future research directions in multimodal logistics data analysis, including the integration of blockchain technology, the utilization of edge computing, and the development of explainable AI models. This paper aims to serve as a valuable resource for researchers and practitioners seeking to leverage multimodal data fusion for improved logistics management, offering insights into the state-of-the-art techniques and future opportunities in this rapidly evolving field.

Multimodal data fusion, e.g., hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion, plays an important role in object recognition and classification tasks. However, existing methods pay little attention to the specificity of HSI spectral channels and the complementarity of HSI and LiDAR spatial information. In addition, the utilized feature extraction modules tend to consider the feature transmission processes among different modalities independently. Therefore, a new data fusion network named AM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> Net is proposed for multimodal data classification; it includes three parts. First, an involution operator slides over the input HSI’s spectral channels, which can independently measure the contribution rate of the spectral channel of each pixel to the spectral feature tensor construction. Furthermore, the spatial information of HSI and LiDAR data is integrated and excavated in an adaptively fused, modality-oriented manner. Second, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spectral-spatial mutual-guided module</i> is designed for the feature collaborative transmission among spectral features and spatial information, which can increase the semantic relatedness connection through adaptive, multiscale, and mutual-learning transmission. Finally, the fused spatial-spectral features are embedded into a classification module to obtain the final results, which determines whether to continue updating the network weights. Experimental evaluations on HSI-LiDAR datasets indicate that AM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> Net possesses a better feature representation ability than the state-of-the-art methods. Additionally, AM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> Net still maintains considerable performance when its input is replaced with multispectral and synthetic aperture radar data. The result indicates that the proposed data fusion framework is compatible with diversified data types.

Estimation of soil moisture content under high maize canopy coverage from UAV multimodal data and machine learning

Advances in multimodality data fusion in neuroimaging

Recognition of microplastic aging features based on multimodal data fusion and attention mechanisms.

Identifying sedimentary facies represents a fundamental aspect of oil and gas exploration. In recent years, geologists have employed deep learning methods to develop comprehensive predictions of sedimentary facies. However, their methods are often constrained to some kind of unimodal data, and the practicality and generalizability of the resulting models are relatively limited. Therefore, based on the characteristics of oilfield data with multiple heterogeneous sources and the difficulty of complementary fusion between data, this paper proposes a sedimentary facies identification technique with multimodal data fusion, which uses multimodal data from core wells, including logging, physical properties, textual descriptions, and core images, to comprehensively predict the sedimentary facies by adopting decision-level feature fusion after predicting different unimodal data separately. The method was applied to a total of 12 core wells in the northwestern margin of the Junggar Basin, China; good results were obtained, achieving an accuracy of over 90% on both the validation and test sets. Using this method, the sedimentary microfacies of a newly drilled core well can be predicted and the interpretation of the sedimentary framework in the well area can be updated in real-time based on data from newly drilled core wells, significantly improving the efficiency and accuracy of oil and gas exploration and development.

Cancer remains a leading cause of morbidity and mortality worldwide, necessitating innovative approaches for early detection and effective management. Predictive analytics, powered by machine learning (ML) and artificial intelligence (AI), offers a transformative potential in cancer care by integrating multi-source health data for early diagnosis, risk assessment, and personalized treatment. This presents a predictive analytics framework that leverages diverse data sources, including electronic health records (EHRs), genomic and biomarker data, medical imaging, wearable sensors, and lifestyle/environmental factors. By combining these heterogeneous data streams, AI models can identify high-risk individuals, detect cancer at its earliest stages, and optimize treatment strategies based on patient-specific insights. The framework employs advanced ML techniques, such as supervised learning models (Random Forests, Support Vector Machines), deep learning architectures (Convolutional Neural Networks, Recurrent Neural Networks), and Natural Language Processing (NLP) for analyzing clinical notes. Federated learning is also explored as a privacy-preserving approach to facilitate secure data sharing across institutions while maintaining compliance with regulatory standards such as HIPAA and GDPR. The integration of predictive analytics with telemedicine platforms and AI-driven clinical decision support systems further enhances real-time monitoring and personalized patient care. Despite its promise, challenges such as data privacy concerns, heterogeneity in multi-source data integration, model interpretability, and computational scalability must be addressed to ensure the effective implementation of AI-driven cancer diagnostics. Future advancements in Explainable AI (XAI), blockchain for secure data exchange, and multi-modal data fusion will further strengthen predictive analytics in oncology. This study underscores the critical role of big data and AI in revolutionizing cancer detection and management, paving the way for improved patient outcomes and more efficient healthcare delivery.

This literature review examines the transformative role of machine learning (ML) and deep learning (DL) in enhancing optical spectroscopy for breast cancer diagnosis. By synthesizing advancements from peer-reviewed studies (2015-2025), we evaluate how ML/DL integration improves the detection of malignancy-associated biochemical changes, enabling noninvasive, rapid, and accurate differentiation between healthy and cancerous tissues. This review highlights key spectroscopic modalities, such as Raman, fluorescence, diffusive optical spectroscopy (DOS), and photoacoustic spectroscopy (PAS), and their integration with AI-driven models, such as convolutional neural networks (CNNs), support vector machines (SVMs), and logistic regression. These techniques achieve diagnostic accuracies of up to 94% in subtype classification (e.g., luminal A, HER2-positive) by analyzing spectral biomarkers such as hemoglobin, lipids, and collagen. Challenges such as data variability, model interpretability, and clinical integration barriers are critically assessed. These findings underscore the potential of ML/DL-enhanced spectroscopy to standardize diagnostics, reduce unnecessary biopsies, and personalize treatment monitoring. Future directions emphasize the need for explainable AI (XAI), multimodal data fusion, and large-scale, diverse datasets to bridge translational gaps. By addressing technical, ethical, and regulatory hurdles, this integration promises to advance early detection, improve clinical outcomes, and reshape precision oncology.

The value of artificial intelligence combined with multimodal data analysis in tumor immunotherapy and targeted therapy.

Monocrystalline silicon is an important raw material in the semiconductor and photovoltaic industries. In the Czochralski (CZ) method of growing monocrystalline silicon, various factors may cause node loss and lead to the failure of crystal growth. Currently, there is no efficient method to detect the node loss of monocrystalline silicon at industrial sites. Therefore, this paper proposed a monocrystalline silicon node-loss detection method based on multimodal data fusion. The aim was to explore a new data-driven approach for the study of monocrystalline silicon growth. This article first collected the diameter, temperature, and pulling speed signals as well as two-dimensional images of the meniscus. Later, the continuous wavelet transform was used to preprocess the one-dimensional signals. Finally, convolutional neural networks and attention mechanisms were used to analyze and recognize the features of multimodal data. In the article, a convolutional neural network based on an improved channel attention mechanism (ICAM-CNN) for one-dimensional signal fusion as well as a multimodal fusion network (MMFN) for multimodal data fusion was proposed, which could automatically detect node loss in the CZ silicon single-crystal growth process. The experimental results showed that the proposed methods effectively detected node-loss defects in the growth process of monocrystalline silicon with high accuracy, robustness, and real-time performance. The methods could provide effective technical support to improve efficiency and quality control in the CZ silicon single-crystal growth process.

Multi-modal data cross-domain fusion network for gearbox fault diagnosis under variable operating conditions