A multi-modal health data fusion and analysis method based on body sensor network

The value of artificial intelligence combined with multimodal data analysis in tumor immunotherapy and targeted therapy.

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

Multi-modal fusion model for Time-Varying medical Data: Addressing Long-Term dependencies and memory challenges in sequence fusion.

This review paper provides a comprehensive overview of logistics behavior analysis and predictive modeling methods based on multimodal data fusion. The increasing complexity of modern logistics necessitates advanced analytical techniques to optimize operations, enhance efficiency, and improve decision-making. Multimodal data fusion, which integrates diverse data sources such as GPS tracking, sensor data, transaction records, and weather information, offers a powerful approach to understanding and predicting logistics behavior. This paper explores the historical evolution of logistics analytics, focusing on the shift from traditional statistical methods to contemporary machine learning techniques. Core themes examined include data acquisition and preprocessing, feature engineering for logistics applications, and the application of various predictive models such as regression models, time series analysis, and deep learning algorithms. A comparative analysis highlights the strengths and weaknesses of different modeling approaches, addressing challenges related to data heterogeneity, scalability, and real-time processing. Furthermore, the review identifies emerging trends and future research directions in multimodal logistics data analysis, including the integration of blockchain technology, the utilization of edge computing, and the development of explainable AI models. This paper aims to serve as a valuable resource for researchers and practitioners seeking to leverage multimodal data fusion for improved logistics management, offering insights into the state-of-the-art techniques and future opportunities in this rapidly evolving field.

In response to the issue of low positioning accuracy and insufficient robustness in small UAVs (unmanned aerial vehicle) caused by sensor noise and cumulative motion errors during flight in complex environments, this paper proposes a multisource, multimodal data fusion method. Initially, it employs a multimodal data fusion of various sensors, including GPS (global positioning system), an IMU (inertial measurement unit), and visual sensors, to complement the strengths and weaknesses of each hardware component, thereby mitigating motion errors to enhance accuracy. To mitigate the impact of sudden changes in sensor data, a high-fidelity UAV model is established in the digital twin based on the real UAV parameters, providing a robust reference for data fusion. By utilizing the extended Kalman filter algorithm, it fuses data from both the real UAV and its digital twin, and the filtered positional information is fed back into the control system of the real UAV. This enables the real-time correction of UAV positional deviations caused by sensor noise and environmental disturbances. The multisource, multimodal fusion Kalman filter method proposed in this paper significantly improves the positioning accuracy of UAVs in complex scenarios and the overall stability of the system. This method holds significant value in maintaining high-precision positioning in variable environments and has important practical implications for enhancing UAV navigation and application efficiency.

Language understanding is multimodal. During human communication, messages are conveyed not only by words in textual form, but also through speech patterns, gestures or facial emotions of the speakers. Therefore, it is crucial to fuse information from different modalities to achieve a joint comprehension. With the rapid progress in the deep learning field, neural networks have emerged as the most popular approach for addressing multimodal data fusion [1, 6, 7, 12]. While these models can effectively combine multimodal features by learning from data, they nevertheless lack an explicit exhibition of how different modalities are related to each other, due to the inherent low interpretability of neural networks [2]. In the meantime, Quantum Theory (QT) has given rise to principled approaches for incorporating interactions between textual features into a holistic textual representation [3, 5, 8, 10], where the concepts of superposition andentanglement have been universally exploited to formulate interactions. The advantages of those models in capturing complicated correlations between textual features have been observed. We hereby propose the research on quantum-inspired multimodal data fusion, claiming that the limitation of multimodal data fusion can be tackled by quantum-driven models. In particular, we propose to employ superposition to formulate intra-modal interactions while the interplay between different modalities is expected to be captured by entanglement measures. By doing so, the interactions within multimodal data may be rendered explicitly in a unified quantum formalism, increasing the performance and interpretability for concrete multimodal tasks. It will also expand the application domains of quantum theory to multimodal tasks where only preliminary efforts have been made [11]. We therefore aim at answering the following research question: RQ. Can we fuse multimodal data with quantum-inspired models? To answer this question, we propose to fuse multimodal data with complex-valued neural networks, motivated by the theoretical link between neural networks and quantum theory [4] and advances in complex-valued neural networks [9]. Our model begins with a separate complex-valued embedding learned for each unimodal data based on the existing works [5, 10] which inherently assumes superposition between intra-modal features. Then we construct a many-body system in entangled state for multimodal data, where cross-modality interactions are naturally reflected by entanglement measures. Quantum measurement operators are applied to the entanglement state to address a concrete multimodal task at hand. The whole process is instrumented by a complex-valued neural network, which is able to learn how multimodal features are combined from data, and at the same time explain the combination by means of quantum superposition and entanglement measures. We plan to examine our proposed models on CMU-MOSI [12] and CMU-MOSEI [1] which are benchmarking multimodal sentiment analysis datasets. The dataset targets at classifying sentiment into 2, 5 or 7 classes with the input of textual, visual and acoustic features. We expect to see comparable effectiveness to state-of-the-art models, and we will explore superposition and entanglement measures to better understand the inter-modal interactions.

Supervised learning models require a large-scale dataset for more effective model training. However, a large-scale dataset requires time-consuming data collection and highly complicated preprocessing, posing a significant challenge for researchers. Therefore, creating a large-scale dataset for each practical application is unachievable in a real-world setting. In this case, unsupervised domain adaptation is crucial for practical applications, as it can enable models to enhance their performance on unlabeled data by training on labeled data. For multi-modal egocentric video analysis, some models have used unsupervised domain adaptation and achieved outstanding performance. However, they use either early or late fusion, ignoring the correlation in and between multi-modal inputs. Therefore, this paper investigates different fusion architecture and proposes a cascade attentional fusion to improve the feature representation for unsupervised domain adaptation on multi-modal egocentric video analysis. First, we propose a cascade fusion architecture for increased audio signal reuse. Then, we propose a temporal-spatial attention mechanism for highlighting spatio-temporal feature representations. Third, we propose a novel cascade attentional fusion method for multi-modal egocentric video data fusion by incorporating the architecture and attention model described previously. In addition, we study the ways for integrating different attentions. Finally, we propose an adversarial domain alignment model that incorporates the proposed fusion for unsupervised domain adaptation on multi-modal egocentric video analysis, reaching state-of-the-art performance on the public dataset.

The integration of multimodal data and artificial intelligence (AI) in precision agriculture has the potential to significantly improve crop monitoring, management, and overall agricultural productivity. Precision agriculture is advancing rapidly by utilizing multimodal data and intelligent system design. These help to improve decision-making and increase farming efficiency. The main challenges are the fusion of data, introducing AI-powered analytics, and human-computer interaction (HCI) in the field of smart agricultural services. To address the challenges, we investigate how to merge various forms of data. Also, we studied innovative applications to use multimodal data and an intelligent system design to improve crop monitoring and management. This research shows a novel aspect to utilize and explains the difficulties of working with a variety of data sources, such as satellite images, sensor data, weather reports, and farmer inputs, to design an intelligent system. In addition, this research examines how advanced data fusion techniques bring these disparate data types together to provide accurate, real-time insights that help optimize farming practices. AI analytics enable intelligent decision-making and timely alerts, resulting in improved agricultural practices using an advanced predictive machine learning (ML) model. This study adds to the expanding body of knowledge on precision agriculture by introducing novel methodology and practical applications, opening the door for more sustainable and intelligent farming systems.

Multimodal emotion recognition based on facial videos aims to extract features from different modalities to identify human emotions. The previous work focus on designing various fusion schemes to combine heterogeneous modal data. However, most studies have overlooked the role of different modalities in emotion recognition and have not fully utilized the intrinsic connections between modalities. Furthermore, the multimodal data from facial videos also contain various distractions bad for emotion analysis. How to reduce the impact of distractions and enable a model to mine effective information for emotion recognition from different modalities is still a challenge problem. To address above issue, we propose a SVD-guided multimodal feature fusion method based on facial video for emotion recognition, which uses a hierarchical fusion mechanism and adopts different loss strategies at each level to learn multimodal feature representation. Specifically, we fuse the facial expression and rPPG signal (or Point-of-Gaze) by using the weak supervision strategy and contrastive learning. Subsequently, the fused feature of facial expression and rPPG signal and the fused feature of facial expression and Point-of-Gaze are combined together to construct the unified multimodal feature matrix. Based on this, Singular Value Decomposition (SVD) is used to refine the redundancy information caused by the multimodal fusion and guide the neural network to learn discriminative emotion feature. At the same time, a consistent loss is developed to enhance the multimodal representation. Experiments on three public datasets show that the proposed method achieves better results over the compared methods.

Multimodal single-cell omics technologies enable multiple molecular programs to be simultaneously profiled at a global scale in individual cells, creating opportunities to study biological systems at a resolution that was previously inaccessible. However, the analysis of multimodal single-cell omics data is challenging due to the lack of methods that can integrate across multiple data modalities generated from such technologies. Here, we present Matilda, a multi-task learning method for integrative analysis of multimodal single-cell omics data. By leveraging the interrelationship among tasks, Matilda learns to perform data simulation, dimension reduction, cell type classification, and feature selection in a single unified framework. We compare Matilda with other state-of-the-art methods on datasets generated from some of the most popular multimodal single-cell omics technologies. Our results demonstrate the utility of Matilda for addressing multiple key tasks on integrative multimodal single-cell omics data analysis. Matilda is implemented in Pytorch and is freely available from https://github.com/PYangLab/Matilda.

Coronary heart disease (CHD) is the leading cause of death and disease burden in China and world-wide. The accurate screening for CHD can be of great value to guiding and facilitating the treatment. Traditional methods, such as computed tomography (CT) and coronary computed tomography angiography (CCTA), are costly and harmful to humans. To address these issues, we proposed a multi-modal data fusion method of ultrasonic (US) images and electronic medical records (EMRs) named M-US-EMRs model to automatically screen CHD. Comparing to traditional methods, this model features cheap, harmless to humans, easy to implement and independent of doctors. The experiment result shows that, M-US-EMRs model reached an overall classification AUC of 79.19\(\%\). Furthermore, the interpretable models that we proposed lead to an increased trust from the people who use them. Our project developed effective tools with good performance for CHD screening among a Chinese population that will help to improve the primary prevention and management of cardiovascular disease. Our method lays the groundwork for using automated interpretation to support serial patient tracking and scalable analysis of millions of ultrasonic images and EMRs archived within healthcare systems.KeywordsUltrasound imagesElectronic medical recordsMulti-modalConvolutional neural network

The diversity of network language data forms, including text, images, audio and other modal data, is not only rich in information content, but also contains users' emotional attitudes and semantic intentions. This paper presents a model that incorporates the unique features of text, image, audio and other modal data. The model includes a multimodal emotion analysis module and a multimodal semantic understanding module. Deep learning method is used to analyze emotion and understand semantics, so as to improve the accuracy and robustness of natural language processing (NLP) technology. In the aspect of emotion analysis, the model combines the feature vectors of different modes by feature level fusion method, and uses the attention mechanism model based on Transformer to classify emotions. In the aspect of semantic understanding, the model integrates the deep features of text and image extracted by BERT and ResNet, and carries out cross-modal semantic reasoning through Transformer model. The experimental findings indicate that the multi-modal fusion model outperforms the single-modal model in emotion analysis, achieving an accuracy of 85.2%, 86.7% recall and 85.0% F1 value, which proves the potential of multi-modal data fusion in enhancing the performance of emotion analysis. At the same time, the precision of the model in semantic tag recognition is 79.3%, the recall rate is 76.8%, and the F1 value is 78.0%, which shows the robustness of the model in avoiding wrong classification and identifying related semantic information. This study provides more intelligent and accurate services for social media monitoring, customer feedback analysis, intelligent customer service and other application fields, and offers important insights for future studies in the area of multimodal information processing.

Chemical imaging, such as mass imaging, provides a distribution image of a particular matter and is crucial for analyzing the chemical and physical mechanisms of a sample. However, methods that provide molecular or elemental distribution do not always have sufficiently high spatial resolution to evaluate the nanosized structures in a sample. To address this issue, a multimodal data analysis method was developed by integrating the obtained low spatial resolution chemical images with complementary methods. In this study, the hydrogen distribution of a steel sample was measured using electron stimulated desorption (ESD) and scanning electron microscopy (SEM). ESD provided the time-course images of hydrogen distribution in the steel sample, whereas SEM provided the outline of the steel sample structure. The multimodal images of the same sample were fused, and then all the data were analyzed together to extract detailed physical and chemical information that cannot be observed by only one of the methods. The alignment of the images obtained using different methods was evaluated based on the minimization of each pixel subtraction. Three different data analysis methods, principal component analysis, least absolute shrinkage and selection operator, and autoencoder, are applied to the image fusion dataset of the ESD image and SEM images to help elucidate the hydrogen permeation behavior through the steel structure.

In this article, the Internet of Medical Things (IoMT) framework based on Apache Spark big data processing technology is proposed for real‐time analysis of health data obtained from wireless body area networks (WBANs), which is one of the most important components of IoMT. The proposed framework consists of four layers: data source, data collection, data analytics and visualization. In addition, the proposed IoMT framework is presented with two different disease prediction scenarios, diabetes and heart disease. Diabetes and heart disease prediction processes are carried out using the random forest (RF), logistic regression (LR) and support vector machine (SVM) algorithms belonging to the Apache Spark machine learning library (MLlib). The analysis of health data generated in WBANs takes place in real‐time in the Apache Spark‐based data analytics layer. In this study, the performances of MLlib algorithms in the real‐time model developed for heart and diabetes disease are examined. The SVM algorithm with an accuracy rate of 93.33% for heart disease and the LR algorithm with an accuracy rate of 78.89% for diabetes are found to provide the best performances.

The LiDAR point cloud data and camera images are distorted to a different degree under various severe weather conditions. Due to this, the traditional single modal object detection methods are unable to use the complementary information between different sensors. Consequently, these algorithms are unable to address various issues caused by severe weather conditions. Recently, the multimodal data fusion methods are applied to the road object detection under severe weather conditions. However, the multimodal algorithms suffer from low data alignment accuracy and the inability to suppress the changes in exposure under severe weather conditions. In this work, we propose a new multimodal sensor fusion object detection network. The proposed network effectively overcomes the shortcomings caused by the camera and LiDAR distortions in severe weather conditions and achieves robust environment perception. We propose: 1) point-wise aligned data fusion method based on K-means++ clustering to improve the accuracy of data alignment; 2) implicit feature pyramid network (i-FPN) to fuse the image features for suppressing the distortions caused by the changes in exposure; 3) hybrid attention mechanism (HAM) to deal with the fusion features and improve the adaptability towards different working conditions. We perform experiments using the ONCE and KITTI dataset. The experimental results and analysis show that the proposed method effectively improves the performance of multimodal deep fusion network under both clear and severe weather conditions.