A Survey on Multimodal Deep Learning for Image Synthesis

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

Maturity of fruits significantly affects various areas of the agriculture industry such as the quality assurance of agricultural products, supply chain, and marketing. However, classifying papaya fruit maturity given six ripeness stages with precision remains a challenge since most changes happen inside the fruit rather than the external characteristics, which are quite similar between stages. Using internal properties in classification would require destructive and time-consuming laboratory tests. With the emergence of deep learning and imaging technologies, data with high dimensions, which correlates with internal and external characteristics of an object such as those produced by hyperspectral cameras, can be processed to perform a high-level intelligent classification task without impairing the fruit. In this paper, we present an AI-derived non-destructive approach that utilizes hyperspectral and visible-light images in estimating the papaya fruit maturity stage and implements multimodality via late fusion of imaging-specific networks. The proposed multimodal architecture is composed of imaging-specific deep convolutional neural networks as base learners and a meta-learner that executes late fusion of the dual unimodal networks. Multiclass logistic regression and averaging are explored as the meta-learners of the multimodal fused network that generates the final classifications. Experimental results of the proposed multimodal-late fused models are compared with the multimodal-feature concatenation approach for estimation of papaya fruit maturity, and our proposed model framework obtained an improved F1-score of up to 0.97. This indicates that multimodal-late fused architecture and multimodal imaging systems have great potential for agricultural and other industrial applications.

This paper presents an approach for adaptive scheduling and robustness optimization in global logistics networks by integrating multimodal deep reinforcement learning with Internet of Things (IoT) technologies. We propose an integrated framework comprising a multimodal data fusion mechanism that synthesizes heterogeneous IoT sensor data, historical records, and contextual information; an adaptive deep reinforcement learning architecture that generates dynamic scheduling policies; and a multi-objective robust optimization method that balances operational efficiency with system resilience. The framework addresses key challenges in global logistics including demand volatility, transportation disruptions, and environmental uncertainties. Comprehensive experiments conducted on real-world logistics datasets demonstrate that our approach outperforms traditional methods with an 18.7% reduction in operational costs, 12.4% improvement in service levels, and significantly enhanced robustness under various disruption scenarios. The proposed method maintains 83% performance stability during complex disruptions compared to 51–72% for alternative approaches, while keeping computational requirements feasible for practical deployment. This research demonstrates potential contributions to AI-driven logistics operations management by showing improved supply chain performance through multimodal learning and robust optimization techniques.

This paper provides a comprehensive review of multimodal deep learning models that utilize conversational data to detect mental health disorders. In addition to discussing models based on the Transformer, such as BERT (Bidirectional Encoder Representations from Transformers), this paper addresses models that existed prior to the Transformer, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The paper covers the application of these models in the construction of multimodal deep learning systems to detect mental disorders. In addition, the difficulties encountered by multimodal deep learning systems are brought up. Furthermore, the paper proposes research directions for enhancing the performance and robustness of these models in mental health applications. By shedding light on the potential of multimodal deep learning in mental health care, this paper aims to foster further research and development in this critical domain.

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

Rapid collection of data sources, varying in volume and structure poses a challenge for scientists to establish a practical approach to manipulating heterogeneous data sources. A multimodal learning and an integrated analysis make it possible to extract much worthwhile information from a collection of multiple simple raw data. Therefore, data integration can lead to a more reliable and robust result. High-throughput sequencing technologies, especially next-generation sequencing, leave us with multi-platform genomic data such as gene expression, SNP, CNV, DNA methylation, and miRNA expression. In this paper, we represented a multimodal deep neural network to exploit the mutual information between three different modalities to classify breast cancer patients into two groups based on their survival rate. Experimental results indicate that our method improves the classification accuracy and performs better on imbalanced data compared to the other single-modal state-of-the-art methods.

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

Mental illness, particularly depression and anxiety, is a leading cause of global disease burden. Underdiagnosis is common due to misperceptions and negative stigma around mental health, limited resources, and self-reporting bias. Newer multimodal deep learning (MDL) frameworks have demonstrated the ability to distill behavioral, linguistic, and physiological signals pertaining to mental health from a number of data streams. However, uptake in clinical practice has been limited partly due to lack of transparency in how the models reach their conclusions. This study proposes a multimodal deep learning framework for the automatic early detection of anxiety and depression from text, audio and video signals with a special focus on Explainable AI(XAI). Basing the research on the benchmark datasets DAIC-WOZ, E-DAIC, and eRisk, the model outperformed unimodal baselines, and delivered clinically meaningful results that were interpretable. The research shows that leveraging explanatory artificial intelligence with MDL frameworks can create a more reliable and transparent AI-based screening tool for mental health problems.

Owing to the limitations of testing methods and the inaccuracy of knowledge, the obtained power transformer information is characterized by fuzziness and randomness, which leads to a lack of accurate description of transformer internal coupling interaction and fault evolution in operation and maintenance departments. Therefore, a thermal state parameter prediction method for a 110 kV transformer based on multi-modal deep learning is designed. The data were resampled and smoothed, and high-frequency noise was removed using a wavelet transform. The running time and spatial temperature rise characteristics of the thermal state parameters of the 110 kV transformer were analyzed, and the variational mode decomposition method was used to decompose the thermal state parameters. By constructing a multimodal deep learning network, abstracted relevant features were extracted from high-dimensional complex input data. By constructing the state equation and measurement equation of the hot spot temperature, a real-time optimal estimation model of the hot spot temperature was established. The extended Kalman filter method was used to build a prediction model for a nonlinear top oil temperature rise system. The experimental results showed that the deviation from the actual value was the least, and the difference between the predicted and measured values of the top oil temperature was relatively small. The prediction time was lower than 0.2s, and the remaining life prediction was correct for 17 units.

Unveiling pedestrian injury risk factors through integration of urban contexts using multimodal deep learning

Real time forecasting of disasters needs to be advanced and easy because with increasing disasters their frequency and severity. Traditional prediction can only be made with traditional disaster prediction methods: numerical weather prediction (NWP) models and remote sensing techniques, which are computationally inefficient, data sparse and cannot adapt to dynamic environmental changes. In order to overcome these limitations, this research presents a Transformer Based Multimodal Deep Learning Model to combine the existing multiple data sources ranging from satellite imagery, IoT sensor networks, meteorological observations etc., to meteorological and social media analytics. The model employs a multimodal fusion strategy, enabling dynamic feature selection and seamless integration of heterogeneous data streams. In contrast to the conventional deep learning techniques, such as CNNs and LSTMs, the transformer based model has excellent ability towards long-range dependency, reducing the latency of light inference and better computational efficiency. The results are proven to be 94% accurate, 91% precise and has 40% reduction in inferencer latency in real time, which makes it suitable for disaster forecasting. The advancement of the multimodal deep learning methodologies presents this research as one which serves to contribute to the AI driven disaster resilience. We will also work on future work in the form of advanced transformer variants, more data integration, and explainable AI (XAI) techniques for model interpretability and scalability. Finding have implications for the transformative potential of AI in climate adaptation and serve as a robust foundation for the next generation early warning systems and climate adaptation disaster risk mitigation across multiple sectors.

As the fashion e-commerce markets rapidly develop, tens of thousands of products are registered daily on e-commerce platforms. Individual sellers register products after setting up a product category directly on a fashion e-commerce platform. However, many sellers fail to find a suitable category and mistakenly register their products under incorrect ones. Precise category matching is important for increasing sales through search optimization and accurate product exposure. However, manually correcting registered categories is time-consuming and costly for platform managers. To resolve this problem, this study proposes a methodology for fashion e-commerce product classification based on multi-modal deep learning and transfer learning. Through the proposed methodology, three challenges in classifying fashion e-commerce products are addressed. First, the issue of extremely biased e-commerce data is addressed through under-sampling. Second, multi-modal deep learning enables the model to simultaneously use input data in different formats, which helps mitigate the impact of noisy and low-quality e-commerce data by providing richer information.Finally, the high computational cost and long training times involved in training deep learning models with both image and text data are mitigated by leveraging transfer learning. In this study, three strategies for transfer learning to fine-tune the image and text modules are presented. In addition, five methods for fusing feature vectors extracted from a single modal into one and six strategies for fine-tuning multi-modal models are presented, featuring a total of 14 strategies. The study shows that multi-modal models outperform unimodal models based solely on text or image. It also suggests the optimal conditions for classifying e-commerce products, helping fashion e-commerce practitioners construct models tailored to their respective business environments more efficiently.

Study question Can a multimodal deep learning approach integrating time-lapse imaging, clinical parameters, and genomic data predict embryo ploidy status in assisted reproductive technology (ART)? Summary answer The multimodal deep learning model can effectively predict ploidy status of embryos using merely embryo imaging. What is known already Current embryo selection relies on subjective morphological assessment or invasive preimplantation genetic testing for aneuploidy (PGT-A). Although PGT-A can diagnose embryo ploidy status and is claimed to enhance pregnancy rates, it presents several limitations including invasive biopsy, high cost, and potentially inaccuracies in detecting mosaicism. Recent artificial intelligence (AI) models show promise but primarily focus on viability prediction rather than genome-wide chromosomal analysis. Despite advances in time-lapse imaging and AI, no validated model currently integrates multimodal data for non-invasive ploidy prediction. Furthermore, existing methods lack the capability to distinguish between meiotic and mitotic chromosomal abnormalities, crucial for understanding developmental potential. Study design, size, duration We retrospectively analyzed 1,400 embryos from PGT cycles (2022-2024) using a stratified random sampling approach. We used ResNet152 as backbone to develop the model, using a 70/20/10 split for training, validation, and testing. Multiple logistic regression and bootstrapping (1000 iterations) were performed for statistical validation. Model performance was assessed using ROCAUC analysis. Participants/materials, setting, methods We included embryos from patients undergoing PGT at our center, cultured in EmbryoScope® time-lapse incubators. Imaging frames were extracted at 2-hour intervals. Morphokinetic parameters (from fertilization to blastocyst formation) and imaging texture features, e.g., density were analyzed. Chromosomal status was determined via whole-genome sequencing of trophectoderm biopsies using haplarithmisis. We then applied class activation mapping (CAM) heatmaps to visualize pixel regions predictive of ploidy outcomes for model interpretation. Main results and the role of chance The multimodal model achieved a ploidy prediction ROCAUC of 0.72, significantly outperforming morphology-only model (ROCAUC: 0.67, p < 0.001). Texture analysis revealed significant correlations between density patterns and ploidy status (Spearman’s ρ = 0.68, p < 0.001). Texture feature scores were significantly different across embryo development process, such as key indicators at fertilization, day3, and day5. CAM heatmap analysis identified inner cell mass region as the strongest morphological predictor. Limitations, reasons for caution Our study has limitations due to its retrospective nature and the single-center design, which may restrict generalizability. Selection bias may occur as we only included embryos progressing to biopsy. External validation across different laboratories and diverse datasets is needed to confirm these findings. Wider implications of the findings Our method demonstrated the potential to reduce reliance on invasive PGT-A, mitigating procedural risks and costs. By enabling non-invasive euploid embryo selection, we aim to enhance the accessibility to precision reproductive care and refine transfer strategies for chromosomal abnormal embryos. Trial registration number No

BackgroundElectroencephalography (EEG) has been widely used to measure brain activity, but its potential to generate accurate images from neural signals remains a challenge. Most EEG-decoding research has focused on tasks such as motor imagery, emotion recognition, and brain wave classification, which involve EEG signal analysis and classification. Some studies have explored the correlation between EEG and images, primarily focusing on EEG-image pair classification or transformation. However, EEG-based image generation remains underexplored.ObjectiveThe primary goal of this study was to extend EEG-based classification to image generation, addressing the limitations of previous methods and unlocking the full potential of EEG for image synthesis. To achieve more meaningful EEG-to-image generation, we developed a novel framework, Neural-Cognitive Multimodal EEG-Informed Image (NECOMIMI), which was specifically designed to generate images directly from EEG signals.MethodsWe developed a 2-stage NECOMIMI method, which integrated the novel Neural Encoding Representation Vectorizer (NERV) EEG encoder that we designed with a diffusion-based generative model. The Category-Based Assessment Table (CAT) score was introduced to evaluate the semantic quality of EEG-generated images. In addition, the ThingsEEG dataset was used to validate and benchmark the CAT score, providing a standardized measure for assessing EEG-to-image generation performance.ResultsThe NERV EEG encoder achieved state-of-the-art performance in several zero-shot classification tasks, with an average accuracy of 94.8% (SD 1.7%) in the 2-way task and 86.8% (SD 3.4%) in the 4-way task, outperforming models such as Natural Image Contrast EEG, Multimodal Similarity-Keeping Contrastive Learning, and Adaptive Thinking Mapper ShallowNet. This highlighted its superiority as a feature extraction tool for EEG signals. In a 1-stage image generation framework, EEG embeddings often resulted in abstract or generalized images such as landscapes instead of specific objects. Our proposed 2-stage NECOMIMI architecture effectively extracted semantic information from noisy EEG signals, showing its ability to capture and represent underlying concepts derived from brain wave activity. We further conducted a perturbation study to test whether the model overly depended on visual cortex EEG signals for scene-based image generation. The perturbation of visual cortex EEG channels led to a notable increase in Fréchet inception distance scores, suggesting that our model relied heavily on posterior brain signals to generate semantically coherent images.ConclusionsNECOMIMI demonstrated the potential of EEG-to-image generation, revealing the challenges of translating noisy EEG data into accurate visual representations. The novel NERV EEG encoder for multimodal contrastive learning reached state-of-the-art performance both on n-way zero-shot and EEG-informed image generation. The introduction of the CAT score provided a new evaluation metric, paving the way for future research to refine generative models. In addition, this study highlighted the significant clinical potential of EEG-to-image generation, particularly in enhancing brain-machine interface systems and improving quality of life for individuals with motor impairments.

Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently. Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance. We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans. Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models. Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.