A deep learning-based multimodal data fusion algorithm for predicting checkpoint inhibitors related pneumonia

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

Sparsity-based and deep learning-based image reconstruction algorithms are two promising approaches to accelerate the image acquisition process for localization-based super-resolution microscopy, by allowing a higher density of fluorescing emitters to be imaged in a single frame. Despite the surging popularity, a comprehensive parametric study guiding the practical applications of sparsity-based and deep learning-based image reconstruction algorithms is yet to be conducted. In this study, we examined the performance of sparsity- and deep learning-based algorithms in reconstructing super-resolution images using simulated fluorescent microscopy images. The simulated images were synthesized with varying levels of sparsity and connectivity. We found the deep learning-based VDSR recovers image faster, with a higher recall rate and localization accuracy. The sparsity-based SPIDER recovers more zero pixels truthfully. We also compared the two algorithms using images acquired from a real super-resolution experiment, yielding results agreeing with the results from the evaluation using simulated images. We concluded that VDSR is preferable when accurate emitter localization is needed while SPIDER is more suitable when evaluation of the number of emitters is critical.

IntroductionWe investigated the impact of ascites and peritoneal metastatic (PM) lesion volumes, measured by deep learning-based algorithm, on survival outcomes in advanced epithelial ovarian cancer.MethodsApplying our newly developed deep learning-based...

Multi-modal fusion model for Time-Varying medical Data: Addressing Long-Term dependencies and memory challenges in sequence fusion.

The impact of partner interaction on brief social buffering in adolescent female rats as analyzed by deep learning-based object detection algorithms.

Multi-modal data fusion has recently been shown promise in classification tasks in remote sensing. Optical data and radar data, two important yet intrinsically different data sources, are attracting more and more attention for potential data fusion. It is already widely known that, a machine learning based methodology often yields excellent performance. However, the methodology relies on a large training set, which is very expensive to achieve in remote sensing. The semi-supervised manifold alignment (SSMA), a multi-modal data fusion algorithm, has been designed to amplify the impact of an existing training set by linking labeled data to unlabeled data via unsupervised techniques. In this paper, we explore the potential of SSMA in fusing optical data and polarimetric SAR data, which are multi-sensory data sources. Furthermore, we propose a MAPPER-induced manifold alignment (MIMA) for semi-supervised fusion of multi-sensory data sources. Our proposed method unites SSMA with MAPPER, which is developed from the emerging topological data analysis (TDA) field. To our best knowledge, this is the first time that SSMA has been applied on fusing optical data and SAR data, and also the first time that TDA has been applied in remote sensing. The conventional SSMA derives a topological structure using k-nearest-neighbor (kNN), while MIMA employs MAPPER, which considers the field knowledge and derives a novel topological structure through the spectral clustering in a data-driven fashion. Experiment results on data fusion with respect to land cover land use classification and local climate zone classification suggest superior performance of MIMA.

With the quick development of smart tiny sensors, gait analysis (GA) methods based on wearable devices have become more popular recently. However, most existing wearable GA methods focus on data analysis from inertial sensors. This paper firstly proposes a novel system based on low-cost, wearable and multimodal sensors for GA, which combines inertial sensor and microphone sensor. The system is introduced by including the wearable multimodal prototype device, related software program, data collection procedure, etc. Secondly, based on the system, a freely available database (ICT_Gait) is published by including the details of setup method and spatial-temporal data label method. This database provides both data from inertial sensor and microphone sensor on two feet from 15 healthy subjects. There are totally 240 data sequences and 1732 steps from three various collection strategies. And this is the first database for multimodal and wearable GA as we know. Finally, based on the system and database, a preliminary research is conducted for footstep detection — the basic and key process for GA. And a coarse-to-fine multimodal fusion algorithm is presented for footstep detection. Experimental results show it achieves average 99.72% for precision rate and 97.83% for recall rate using multimodal fusion algorithm, which presents the effectiveness of multimodal fusion for wearable GA.

This paper develops a wearable sport activity classification system and its associated deep learning-based sport activity classification algorithm for accurately recognizing sport activities. The proposed wearable system used two wearable inertial sensing modules worn on athletes’ wrist and ankle to collect sport motion signals and utilized a deep convolutional neural network (CNN) to extract the inherent features from the spectrograms of the short-term Fourier transform (STFT) of the sport motion signals. The wearable inertial sensing module is composed of a microcontroller, a triaxial accelerometer, a triaxial gyroscope, an RF wireless transmission module, and a power supply circuit. All ten participants wore the two wearable inertial sensing modules on their wrist and ankle to collect motion signals generated by sport activities. Subsequently, we developed a deep learning-based sport activity classification algorithm composed of sport motion signal collection, signal preprocessing, sport motion segmentation, signal normalization, spectrogram generation, image mergence/resizing, and CNN-based classification to recognize ten types of sport activities. The CNN classifier consisting of two convolutional layers, two pooling layers, a fully-connected layer, and a softmax layer can be used to divide the sport activities into table tennis, tennis, badminton, golf, batting baseball, shooting basketball, volleyball, dribbling basketball, running, and bicycling, respectively. Finally, the experimental results show that the proposed wearable sport activity classification system and its deep learning-based sport activity classification algorithm can recognize 10 sport activities with the classification rate of 99.30%.

Due to the lack of consistent findings across different modalities, the neurobiological underpinning of bipolar disorder (BD) remains elusive. This study aims to employ a multimodal fusion algorithm, integrating multimodal imaging data, to unravel the neurobiological underpinning of BD.A data-driven multimodal fusion algorithm was utilized to analyze covariant patterns across modalities in a cohort of 125 BD patients and 113 healthy controls (HCs). The study focused on fusing regional homogeneity (ReHo), gray matter volume (GMV), and fractional anisotropy (FA) derived from MRI scans to generate group-discriminative joint independent components (jIC).That differentiated BD patients from HCs across three modalities. An inverse functional pattern was observed in the default mode network (DMN) and sensorimotor network (SMN) in BD patients, characterized by increased ReHo in the DMN and decreased ReHo in the SMN compared to healthy individuals. This inverse pattern was also mirrored in GMV, showing increase in the DMN and decreases in the SMN. Meanwhile, significant functional hyperactivation coupled with decreased structural volume in the precuneus underscores its role in cognitive function in BD.Multimodal neuroimaging fusion provides a comprehensive understanding in pathophysiology of BD, offering valuable insights that could be pivotal in advancing the diagnosis and treatment of BD.

This paper discusses the application of machine learning algorithms in medical diagnosis, aiming to explore their impact on the accuracy and efficiency of disease diagnosis. Through case analysis and literature review, the paper elaborates on the application of machine learning algorithms in medical diagnosis, including multimodal data fusion, automated diagnosis, and deep learning algorithms. The research indicates that machine learning algorithms can improve the accuracy and efficiency of disease diagnosis through learning and analysis of large amounts of data. While machine learning algorithms have great potential in medical diagnosis, it is essential to ensure data quality, privacy, and algorithm transparency. Interdisciplinary collaboration is recommended to promote sustainable development in healthcare.

New method for modeling digital twin behavior perception of cows: Cow daily behavior recognition based on multimodal data

This paper presents a multi-mode fusion algorithm for detection and tracking of dim, point-like target. The key contribution of this paper includes the effective fusion approach to harvest the advantages and complement the disadvantages of various algorithms using conditional voting. From qualitative analysis, these algorithms are separated into two classes, i.e. main and supporting algorithms. In the multi-mode fusion algorithm high confidence is placed on the main algorithms with supporting algorithms used to further reduce the false alarm. The main algorithms trigger a voting process and detection is confirmed true if any of the supporting algorithms report detection. The multi-modal fusion algorithm has lower false alarm and moderate true detection rate compared to any individual algorithm namely, Triple Temporal Filter, Frame Differencing, Continuous Wavelet Transform, Max-median and 2-D Mexican hat filter. Besides, a novel variability filter is proposed to remove strong glint thus reduces false alarm. Kalman filter is used to track the detected targets. A novel track decision algorithm to continue or terminate the track when target disappears is proposed. Prior knowledge of target in Kalman filter is fed forward to an Adaptive 3-D Matched filter to improve the performance. Three sets of real-world infrared image sequences with very different background and target characteristics were used to test the robustness of the multi-modal fusion algorithm. The algorithm performs satisfactorily in all the image sequences. Video clips will also be presented.© (2003) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Language understanding is multimodal. During human communication, messages are conveyed not only by words in textual form, but also through speech patterns, gestures or facial emotions of the speakers. Therefore, it is crucial to fuse information from different modalities to achieve a joint comprehension. With the rapid progress in the deep learning field, neural networks have emerged as the most popular approach for addressing multimodal data fusion [1, 6, 7, 12]. While these models can effectively combine multimodal features by learning from data, they nevertheless lack an explicit exhibition of how different modalities are related to each other, due to the inherent low interpretability of neural networks [2]. In the meantime, Quantum Theory (QT) has given rise to principled approaches for incorporating interactions between textual features into a holistic textual representation [3, 5, 8, 10], where the concepts of superposition andentanglement have been universally exploited to formulate interactions. The advantages of those models in capturing complicated correlations between textual features have been observed. We hereby propose the research on quantum-inspired multimodal data fusion, claiming that the limitation of multimodal data fusion can be tackled by quantum-driven models. In particular, we propose to employ superposition to formulate intra-modal interactions while the interplay between different modalities is expected to be captured by entanglement measures. By doing so, the interactions within multimodal data may be rendered explicitly in a unified quantum formalism, increasing the performance and interpretability for concrete multimodal tasks. It will also expand the application domains of quantum theory to multimodal tasks where only preliminary efforts have been made [11]. We therefore aim at answering the following research question: RQ. Can we fuse multimodal data with quantum-inspired models? To answer this question, we propose to fuse multimodal data with complex-valued neural networks, motivated by the theoretical link between neural networks and quantum theory [4] and advances in complex-valued neural networks [9]. Our model begins with a separate complex-valued embedding learned for each unimodal data based on the existing works [5, 10] which inherently assumes superposition between intra-modal features. Then we construct a many-body system in entangled state for multimodal data, where cross-modality interactions are naturally reflected by entanglement measures. Quantum measurement operators are applied to the entanglement state to address a concrete multimodal task at hand. The whole process is instrumented by a complex-valued neural network, which is able to learn how multimodal features are combined from data, and at the same time explain the combination by means of quantum superposition and entanglement measures. We plan to examine our proposed models on CMU-MOSI [12] and CMU-MOSEI [1] which are benchmarking multimodal sentiment analysis datasets. The dataset targets at classifying sentiment into 2, 5 or 7 classes with the input of textual, visual and acoustic features. We expect to see comparable effectiveness to state-of-the-art models, and we will explore superposition and entanglement measures to better understand the inter-modal interactions.

It is a challenging problem to accurately detect overlapping defects. This paper proposes a system that can obtain optical and electromagnetic signals synchronously. It contains an industrial camera, a chromatic confocal displacement sensor (CCDS) and an eddy current testing (ECT) probe with high-resolution array tunneling magnetoresistance (TMR) sensors. A multimodal fusion algorithm is proposed to combine the advantages of the optical testing and ECT. The fusion of the visual image and CCDS measurement results in morphology of the surface defects. To recognize the overlapping buried defect, a finite element method model is developed to estimate the surface defect signal using parameters from the optical measurements, and then the surface defect signal is subtracted from the experimental image, based on which the overlapping defects are precisely detected. This method can be widely used in industry to comprehensively assess the health and integrity of structures.

Using multimodal neuroimaging data to characterize brain network is currently an advanced technique for Alzheimer’s disease(AD) Analysis. Over recent years the neuroimaging community has made tremendous progress in the study of resting-state functional magnetic resonance imaging (rs-fMRI) derived from blood-oxygen-level-dependent (BOLD) signals and Diffusion Tensor Imaging (DTI) derived from white matter fiber tractography. However, Due to the heterogeneity and complexity between BOLD signals and fiber tractography, Most existing multimodal data fusion algorithms can not sufficiently take advantage of the complementary information between rs-fMRI and DTI. To overcome this problem, a novel Hypergraph Generative Adversarial Networks (HGGAN) is proposed in this paper, which utilizes Interactive Hyperedge Neurons module (IHEN) and Optimal Hypergraph Homomorphism algorithm (OHGH) to generate multimodal connectivity of Brain Network from rs-fMRI combination with DTI. To evaluate the performance of this model, We use publicly available data from the ADNI database to demonstrate that the proposed model not only can identify discriminative brain regions of AD but also can effectively improve classification performance.