Chapter 2 - Deep Learning for Multimodal Data Fusion

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

In recent years, multitask learning has turned out to be of great success in various applications. Though single task model training has promised great results throughout these years, it ignores valuable information which might help in estimating better learning parameters. By virtue of learning-related tasks, multitask learning has been able to generalize the models even better. In this paper, we try to enhance the feature mapping of multitask deep learning models by sharing features among related tasks and inductive transfer learning. We have explored the learning of task relationship among various tasks for acquiring more benefits from multitask learning. The proposed enhanced model is compared with two state-of-the-art multitask deep learning models, namely Hyperface and FaceNet. The results show better performance of the proposed model in predicting age, gender and ethnicity on the UTK face dataset.KeywordsNeural networksMultitask deep learningDeep learningRecurrent neural networksConvolution neural networks

Magnetotelluric (MT) forward modeling is a key technique in magnetotelluric sounding, and deep learning has been widely applied to MT forward modeling. In three-dimensional (3-D) problems, although existing methods can predict forward modeling results with high accuracy, they often use multiple networks to simulate multiple forward modeling parameters, resulting in low efficiency. We apply multi-task learning (MTL) to 3-D MT forward modeling to achieve simultaneous inference of apparent resistivity and impedance phase, effectively improving overall efficiency. Furthermore, through comparative analysis of feature map differences in various decoder layers of the network, we identify the optimal branching point for multi-task learning decoders. This enhances the feature extraction capabilities of the network and improves the prediction accuracy of forward modeling parameters. Additionally, we introduce an uncertainty-based loss function to dynamically balance the learning weights between tasks, addressing the shortcomings of traditional loss functions. Experiments demonstrate that compared with single-task networks and existing multi-task networks, the proposed network (MT-FeatureNet) achieves the best results in terms of Structural Similarity Index Measure (SSIM), Mean Relative Error (MRE), and Mean Absolute Error (MAE). The proposed multi-task learning model not only improves the efficiency and accuracy of 3-D MT forward modeling but also provides a novel approach to the design of multi-task learning network structures.

This is an accepted manuscript of an article published by Association for Computational Linguistics in Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019), available online: https://www.aclweb.org/anthology/W19-5119
\n
\nThe accepted version of the publication may differ from the final published version.

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge-it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

Convolutional neural networks (CNN) have been used successfully in solving many challenging visual perception tasks facing mobile robots and self-driving cars. To facilitate deploying such models on embedded hardware onboard mobile robots that have limited resources, multitask learning approaches have become common. In typically used multitask learning, a shared encoder network extracts features from inputs whereas multiple task-specific decoders transform these features into their target output. However, properly combining different tasks’ losses into the final network loss such that each task is making progress learning is a major challenge in these approaches. In this article, we present an innovative approach to extend a typical single-task network with the capability of performing two tasks without multiple decoders, i.e., a single-stream two-task network. The two output tasks are semantic segmentation and monocular depth prediction which are essential tasks in visual perception for autonomous driving. The method is centered on solving semantic segmentation with a regression loss function rather than a classification one. With our approach, we seize multitask learning benefits of reduced overhead and enhanced generalization while alleviating the need to balance different loss functions. Experimental evaluations with baseline single tasks and a multitask network are presented.

Multi-task learning (MTL) is a machine learning paradigm that improves the performance of each task by exploiting useful information contained in multiple related tasks. However, the relatedness of tasks can be exploited by attackers to launch data poisoning attacks, which has been demonstrated a big threat to single-task learning. In this paper, we provide the first study on the vulnerability of MTL. Specifically, we focus on multi-task relationship learning (MTRL) models, a popular subclass of MTL models where task relationships are quantized and are learned directly from training data. We formulate the problem of computing optimal poisoning attacks on MTRL as a bilevel program that is adaptive to arbitrary choice of target tasks and attacking tasks. We propose an efficient algorithm called PATOM for computing optimal attack strategies. PATOM leverages the optimality conditions of the subproblem of MTRL to compute the implicit gradients of the upper level objective function. Experimental results on real-world datasets show that MTRL models are very sensitive to poisoning attacks and the attacker can significantly degrade the performance of target tasks, by either directly poisoning the target tasks or indirectly poisoning the related tasks exploiting the task relatedness. We also found that the tasks being attacked are always strongly correlated, which provides a clue for defending against such attacks.

Recent research in disaster informatics demonstrates a practical and important use case of artificial intelligence to save human lives and suffering during natural disasters based on social media contents (text and images). While notable progress has been made using texts, research on exploiting the images remains relatively under-explored. To advance image-based approaches, we propose MEDIC (https://crisisnlp.qcri.org/medic/index.html), which is the largest social media image classification dataset for humanitarian response consisting of 71,198 images to address four different tasks in a multi-task learning setup. This is the first dataset of its kind: social media images, disaster response, and multi-task learning research. An important property of this dataset is its high potential to facilitate research on multi-task learning, which recently receives much interest from the machine learning community and has shown remarkable results in terms of memory, inference speed, performance, and generalization capability. Therefore, the proposed dataset is an important resource for advancing image-based disaster management and multi-task machine learning research. We experiment with different deep learning architectures and report promising results, which are above the majority baselines for all tasks. Along with the dataset, we also release all relevant scripts (https://github.com/firojalam/medic).

We tackle the problem of multi-task learningMulti-Task Learning (MTL) (MTL) for solving the correlated visual dense prediction tasks using monocular image source. With further restrictions on the model parameters, the soft-shared MTL Convnets (CNN) feature distinct models for each individual task. Hard-sharingHard-sharing-based models, on the other hand, have shared encoders but individual decoders for every task. MTLMulti-Task Learning (MTL) models have demonstrated satisfactory performances for pixel-wise dense prediction tasks like semantic segmentation, surface-normal estimation, and depth estimation from the monocular inputs. In general, they impose two inherent drawbacks: (1) such models have constraints with no leverage on the inter-task knowledge, which hinders the performance boost while jointly trained, and (2) explicitly optimization of each task-specific network is required. Incorporating the abovementioned issues into soft and hard-sharingHard-sharing-based MTL models can immensely enhance MTLMulti-Task Learning (MTL) performance. To that end, we present SD-MTCNN, hard-sharing-based and S3DMT-Net, soft-sharingSoft-sharing-based novel MTL networks. Here, we follow tries to inherit characteristics from deeper CNNConvolutional Neural Network (CNN) layers/feature maps into shallower CNN layers, hence helps in increasing the network's bandwidth. We also utilize the notion of sharing the features of the task-specific encoders, where the task-specific encoder’s feature-maps are communicated amongst each other, aka as cross-task interactions. We examine our self-distilled MTLsMulti-Task Learning (MTL) performance on two different types of visual scenes: Urban (CityScapes and ISPRS) and indoor (NYUv2 and Mini-Taskonomy), and we observe notable gains in all tasks as compared to the referred methodologies.

Semantic segmentation is a fundamental task in image processing and computer vision domains. Therefore, it is a core component of most emerging industry applications such as medical imaging, autonomous driving, and agriculture. The last five years have witnessed a huge growth in the computer vision domain with the introduction of Generative Adversarial Networks (GANs) due to their ability to learn deep representations by a competitive process between two neural networks called the Generator the Discriminator. Neural Architecture Search (NAS) is undergoing a revolution aimed at automating the architectural engineering process. This paper discusses implementing a novel framework for semantic image segmentation called iSegmentor using NAS and GANs. The proposed approach is the first framework that utilizes NAS in GANs for semantic image segmentation to the best of our knowledge. The iSegmentor framework was evaluated using mean Intersection over Union (mIoU) and Mean Pixel Accuracy (MPA) metrics. The iSegmentor was mainly tested and evaluated for the autonomous driving domain. With the analysis of the past five-year literature, Cityscapes and CamVid datasets were chosen. The iSegmentor achieved 81.73 MPA and 71.90 mIoU for the Cityscapes dataset, whereas CamVid achieved 80.41 MPA and 70.63 mIoU. Further, we present future avenues that we identified with the conducted experiments to assist other researchers to extend the research area of NAS in GANs in semantic segmentation.

Objective. Deep learning (DL)-based automated contouring and treatment planning has been proven to improve the efficiency and accuracy of radiotherapy. However, conventional radiotherapy treatment planning process has the automated contouring and treatment planning as separate tasks. Moreover in DL, the contouring and dose prediction tasks for automated treatment planning are done independently.Approach. In this study, we applied the multi-task learning (MTL) approach in order to seamlessly integrate automated contouring and voxel-based dose prediction tasks, as MTL can leverage common information between the two tasks and be able to increase the efficiency of the automated tasks. We developed our MTL framework using the two datasets: in-house prostate cancer dataset and the publicly available head and neck cancer dataset, OpenKBP.Main results. Compared to the sequential DL contouring and treatment planning tasks, our proposed method using MTL improved the mean absolute difference of dose volume histogram metrics of prostate and head and neck sites by 19.82% and 16.33%, respectively. Our MTL model for automated contouring and dose prediction tasks demonstrated enhanced dose prediction performance while maintaining or sometimes even improving the contouring accuracy. Compared to the baseline automated contouring model with the Dice score coefficients of 0.818 for prostate and 0.674 for head and neck datasets, our MTL approach achieved average scores of 0.824 and 0.716 for these datasets, respectively.Significance. Our study highlights the potential of the proposed automated contouring and planning using MTL to support the development of efficient and accurate automated treatment planning for radiotherapy.

Chapter 12 - Semantic scene segmentation for robotics

Purpose To develop and evaluate a novel multitask deep learning framework for automated detection and localization of endoleaks at aortic digital subtraction angiography (DSA) performed during real-world endovascular aneurysm repair (EVAR) procedures for abdominal aortic aneurysm. Materials and Methods This retrospective study analyzed intraoperative aortic DSA images from patients undergoing EVAR (January 2017-December 2021). An expert panel assessed each sequence for endoleaks. Each sequence was processed into three input channels: peak density, time to peak, and area under the time-density curve, generating three two-dimensional perfusion maps per patient. These maps served as input into a convolutional neural network for binary detection (classification) and localization (regression) of endoleaks through multitask learning. Fivefold cross-validation was performed, with patients split 80:20 into training and testing datasets for each fold. Performance metrics included area under the receiver operating characteristic curve, F1 score, precision, and recall and were compared with human experts. Results The study included 220 patients (median age, 74 years [IQR, 68-79]; 181 male). Endoleaks were visible in 111 of 220 (50.5%) patients. The model identified and localized endoleaks with an area under the receiver operating characteristic curve of 0.85 ± 0.0031 (SD), F1 score of 0.78 ± 0.21, 95% precision, and 73% recall. Compared with the procedural team (94% precision, 63% recall), it had higher values in both metrics, with an F1 score within the human observer range (0.75-0.85). Balancing regression and classification by multitask learning delivered optimal results. The interobserver agreement among human experts was moderate (Fleiss κ = 0.404). Conclusion A novel, fully automated deep learning method accurately detected and localized endoleaks at DSA imaging from EVAR procedures. Keywords: Endoleaks, Deep Learning, Digital Subtraction Angiography, Endovascular Aneurysm Repair, Abdominal Aortic Aneurysm, Interventional Suite, Hybrid Operating Room Supplemental material is available for this article. © RSNA, 2025.

Esophageal cancer is a globally significant but understudied type of cancer with high mortality rates. The staging and differentiation of esophageal cancer are crucial factors in determining the prognosis and surgical treatment plan for patients, as well as improving their chances of survival. Endoscopy and histopathological examination are considered as the gold standard for esophageal cancer diagnosis. However, some previous studies have employed deep learning-based methods for esophageal cancer analysis, which are limited to single-modal features, resulting in inadequate classification results. In response to these limitations, multi-modal learning has emerged as a promising alternative for medical image analysis tasks. In this paper, we propose a hierarchical feature fusion network, MM-HiFuse, for multi-modal multitask learning to improve the classification accuracy of esophageal cancer staging and differentiation level. The proposed architecture combines low-level to deep-level features of both pathological and endoscopic images to achieve accurate classification results. The key characteristics of MM-HiFuse include: (i) a parallel hierarchy of convolution and self-attention layers specifically designed for pathological and endoscopic image features; (ii) a multi-modal hierarchical feature fusion module (MHF) and a new multitask weighted combination loss function. The benefits of these features are the effective extraction of multi-modal representations at different semantic scales and the mutual complementarity of the multitask learning, leading to improved classification performance. Experimental results demonstrate that MM-HiFuse outperforms single-modal methods in esophageal cancer staging and differentiation classification. Our findings provide evidence for the early diagnosis and accurate staging of esophageal cancer and serve as a new inspiration for the application of multi-modal multitask learning in medical image analysis. Code is available at https://github.com/huoxiangzuo/MM-HiFuse.

Toxicity is paramount for comprehending compound properties, particularly in the early stages of drug design. Due to the diversity and complexity of toxic effects, it became a challenge to compute compound toxicity tasks. To address this issue, we propose a multimodal representation learning model, termed multimodal graph isomorphism network (MMGIN), to address this challenge for compound toxicity multitask learning. Based on fingerprints and molecular graphs of compounds, our MMGIN model incorporates a multimodal representation learning model to acquire a comprehensive compound representation. This model adopts a two-channel structure to independently learn fingerprint representation and molecular graph representation. Subsequently, two feedforward neural networks utilize the learned multimodal compound representation to perform multitask learning, encompassing compound toxicity classification and multiple compound category classification simultaneously. To test the effectiveness of our model, we constructed a novel data set, termed the compound toxicity multitask learning (CTMTL) data set, derived from the TOXRIC data set. We compare our MMGIN model with other representative machine learning and deep learning models on the CTMTL and Tox21 data sets. The experimental results demonstrate significant advancements achieved by our MMGIN model. Furthermore, the ablation study underscores the effectiveness of the introduced fingerprints, molecular graphs, the multimodal representation learning model, and the multitask learning model, showcasing the model's superior predictive capability and robustness.