A multimodal deep reinforcement learning approach for IoT-driven adaptive scheduling and robustness optimization in global logistics networks

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

Rapid collection of data sources, varying in volume and structure poses a challenge for scientists to establish a practical approach to manipulating heterogeneous data sources. A multimodal learning and an integrated analysis make it possible to extract much worthwhile information from a collection of multiple simple raw data. Therefore, data integration can lead to a more reliable and robust result. High-throughput sequencing technologies, especially next-generation sequencing, leave us with multi-platform genomic data such as gene expression, SNP, CNV, DNA methylation, and miRNA expression. In this paper, we represented a multimodal deep neural network to exploit the mutual information between three different modalities to classify breast cancer patients into two groups based on their survival rate. Experimental results indicate that our method improves the classification accuracy and performs better on imbalanced data compared to the other single-modal state-of-the-art methods.

In recent years, deep learning technology has made breakthroughs in computer vision. After using large-scale data training, the deep neural network represented by GAN is significantly better than previous technologies in image generation, including generating more reasonable, higher-definition, more complex, and more accurate images. With the continuous development of datasets, models, and applications, the fusion of different modal information, including fusion of natural language, semantic layouts, tags, edge maps, and other different modal information, to generate images has become a new demand and challenge. There are related reviews on image generation and multimodal deep learning. However, there has not been a review dedicated to multimodal deep-learning image generation to discuss the current status, existing problems, and challenges of this task. Therefore, this review proposes a survey on multimodal deep learning image generation. It aims to provide readers with an application scenario for multimodal deep learning image generation. Also, it provides readers with new multimodal deep learning image generation technologies, the relevant datasets, evaluation metrics used, and some results comparison. Finally, this article describes some of the challenges and future topics of multimodal deep learning image generation.

This paper provides a comprehensive review of multimodal deep learning models that utilize conversational data to detect mental health disorders. In addition to discussing models based on the Transformer, such as BERT (Bidirectional Encoder Representations from Transformers), this paper addresses models that existed prior to the Transformer, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The paper covers the application of these models in the construction of multimodal deep learning systems to detect mental disorders. In addition, the difficulties encountered by multimodal deep learning systems are brought up. Furthermore, the paper proposes research directions for enhancing the performance and robustness of these models in mental health applications. By shedding light on the potential of multimodal deep learning in mental health care, this paper aims to foster further research and development in this critical domain.

Study question Can a multimodal deep learning approach integrating time-lapse imaging, clinical parameters, and genomic data predict embryo ploidy status in assisted reproductive technology (ART)? Summary answer The multimodal deep learning model can effectively predict ploidy status of embryos using merely embryo imaging. What is known already Current embryo selection relies on subjective morphological assessment or invasive preimplantation genetic testing for aneuploidy (PGT-A). Although PGT-A can diagnose embryo ploidy status and is claimed to enhance pregnancy rates, it presents several limitations including invasive biopsy, high cost, and potentially inaccuracies in detecting mosaicism. Recent artificial intelligence (AI) models show promise but primarily focus on viability prediction rather than genome-wide chromosomal analysis. Despite advances in time-lapse imaging and AI, no validated model currently integrates multimodal data for non-invasive ploidy prediction. Furthermore, existing methods lack the capability to distinguish between meiotic and mitotic chromosomal abnormalities, crucial for understanding developmental potential. Study design, size, duration We retrospectively analyzed 1,400 embryos from PGT cycles (2022-2024) using a stratified random sampling approach. We used ResNet152 as backbone to develop the model, using a 70/20/10 split for training, validation, and testing. Multiple logistic regression and bootstrapping (1000 iterations) were performed for statistical validation. Model performance was assessed using ROCAUC analysis. Participants/materials, setting, methods We included embryos from patients undergoing PGT at our center, cultured in EmbryoScope® time-lapse incubators. Imaging frames were extracted at 2-hour intervals. Morphokinetic parameters (from fertilization to blastocyst formation) and imaging texture features, e.g., density were analyzed. Chromosomal status was determined via whole-genome sequencing of trophectoderm biopsies using haplarithmisis. We then applied class activation mapping (CAM) heatmaps to visualize pixel regions predictive of ploidy outcomes for model interpretation. Main results and the role of chance The multimodal model achieved a ploidy prediction ROCAUC of 0.72, significantly outperforming morphology-only model (ROCAUC: 0.67, p < 0.001). Texture analysis revealed significant correlations between density patterns and ploidy status (Spearman’s ρ = 0.68, p < 0.001). Texture feature scores were significantly different across embryo development process, such as key indicators at fertilization, day3, and day5. CAM heatmap analysis identified inner cell mass region as the strongest morphological predictor. Limitations, reasons for caution Our study has limitations due to its retrospective nature and the single-center design, which may restrict generalizability. Selection bias may occur as we only included embryos progressing to biopsy. External validation across different laboratories and diverse datasets is needed to confirm these findings. Wider implications of the findings Our method demonstrated the potential to reduce reliance on invasive PGT-A, mitigating procedural risks and costs. By enabling non-invasive euploid embryo selection, we aim to enhance the accessibility to precision reproductive care and refine transfer strategies for chromosomal abnormal embryos. Trial registration number No

Mental illness, particularly depression and anxiety, is a leading cause of global disease burden. Underdiagnosis is common due to misperceptions and negative stigma around mental health, limited resources, and self-reporting bias. Newer multimodal deep learning (MDL) frameworks have demonstrated the ability to distill behavioral, linguistic, and physiological signals pertaining to mental health from a number of data streams. However, uptake in clinical practice has been limited partly due to lack of transparency in how the models reach their conclusions. This study proposes a multimodal deep learning framework for the automatic early detection of anxiety and depression from text, audio and video signals with a special focus on Explainable AI(XAI). Basing the research on the benchmark datasets DAIC-WOZ, E-DAIC, and eRisk, the model outperformed unimodal baselines, and delivered clinically meaningful results that were interpretable. The research shows that leveraging explanatory artificial intelligence with MDL frameworks can create a more reliable and transparent AI-based screening tool for mental health problems.

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

Owing to the limitations of testing methods and the inaccuracy of knowledge, the obtained power transformer information is characterized by fuzziness and randomness, which leads to a lack of accurate description of transformer internal coupling interaction and fault evolution in operation and maintenance departments. Therefore, a thermal state parameter prediction method for a 110 kV transformer based on multi-modal deep learning is designed. The data were resampled and smoothed, and high-frequency noise was removed using a wavelet transform. The running time and spatial temperature rise characteristics of the thermal state parameters of the 110 kV transformer were analyzed, and the variational mode decomposition method was used to decompose the thermal state parameters. By constructing a multimodal deep learning network, abstracted relevant features were extracted from high-dimensional complex input data. By constructing the state equation and measurement equation of the hot spot temperature, a real-time optimal estimation model of the hot spot temperature was established. The extended Kalman filter method was used to build a prediction model for a nonlinear top oil temperature rise system. The experimental results showed that the deviation from the actual value was the least, and the difference between the predicted and measured values of the top oil temperature was relatively small. The prediction time was lower than 0.2s, and the remaining life prediction was correct for 17 units.

Unveiling pedestrian injury risk factors through integration of urban contexts using multimodal deep learning

Real time forecasting of disasters needs to be advanced and easy because with increasing disasters their frequency and severity. Traditional prediction can only be made with traditional disaster prediction methods: numerical weather prediction (NWP) models and remote sensing techniques, which are computationally inefficient, data sparse and cannot adapt to dynamic environmental changes. In order to overcome these limitations, this research presents a Transformer Based Multimodal Deep Learning Model to combine the existing multiple data sources ranging from satellite imagery, IoT sensor networks, meteorological observations etc., to meteorological and social media analytics. The model employs a multimodal fusion strategy, enabling dynamic feature selection and seamless integration of heterogeneous data streams. In contrast to the conventional deep learning techniques, such as CNNs and LSTMs, the transformer based model has excellent ability towards long-range dependency, reducing the latency of light inference and better computational efficiency. The results are proven to be 94% accurate, 91% precise and has 40% reduction in inferencer latency in real time, which makes it suitable for disaster forecasting. The advancement of the multimodal deep learning methodologies presents this research as one which serves to contribute to the AI driven disaster resilience. We will also work on future work in the form of advanced transformer variants, more data integration, and explainable AI (XAI) techniques for model interpretability and scalability. Finding have implications for the transformative potential of AI in climate adaptation and serve as a robust foundation for the next generation early warning systems and climate adaptation disaster risk mitigation across multiple sectors.

As the fashion e-commerce markets rapidly develop, tens of thousands of products are registered daily on e-commerce platforms. Individual sellers register products after setting up a product category directly on a fashion e-commerce platform. However, many sellers fail to find a suitable category and mistakenly register their products under incorrect ones. Precise category matching is important for increasing sales through search optimization and accurate product exposure. However, manually correcting registered categories is time-consuming and costly for platform managers. To resolve this problem, this study proposes a methodology for fashion e-commerce product classification based on multi-modal deep learning and transfer learning. Through the proposed methodology, three challenges in classifying fashion e-commerce products are addressed. First, the issue of extremely biased e-commerce data is addressed through under-sampling. Second, multi-modal deep learning enables the model to simultaneously use input data in different formats, which helps mitigate the impact of noisy and low-quality e-commerce data by providing richer information.Finally, the high computational cost and long training times involved in training deep learning models with both image and text data are mitigated by leveraging transfer learning. In this study, three strategies for transfer learning to fine-tune the image and text modules are presented. In addition, five methods for fusing feature vectors extracted from a single modal into one and six strategies for fine-tuning multi-modal models are presented, featuring a total of 14 strategies. The study shows that multi-modal models outperform unimodal models based solely on text or image. It also suggests the optimal conditions for classifying e-commerce products, helping fashion e-commerce practitioners construct models tailored to their respective business environments more efficiently.

This study examines how CEO personality traits influence IPO risk disclosure quality. We employ a novel multimodal deep learning approach to measure CEO personality traits using audio-visual data from IPO roadshow presentations. Our methodology integrates visual and audio data to capture both verbal and non-verbal behavioral cues, providing a more comprehensive assessment of personality traits than traditional single-modal approaches. Using a manually collected dataset of 866 IPO firms from China's ChiNext and STAR Markets between 2019 and 2024, we find that openness, conscientiousness, and extraversion are positively associated with risk disclosure quality, while agreeableness and neuroticism show negative associations. These relationships are moderated by underwriter reputation, particularly strengthening the positive effects of openness and conscientiousness. This study contributes to the literature by introducing a novel methodological approach to measuring CEO personality traits through multimodal deep learning analysis, extending our understanding of IPO risk disclosure determinants, and demonstrating the effectiveness of multimodal data analysis in organizational research.

Fluorine (F) substitution is a common method of drug discovery and development. However, there are no accurate approaches available for predicting the bioactivity changes after F-substitution, as the effect of substitution on the interactions between compounds and proteins (CPI) remains a mystery. In this study, we constructed a data set with 111,168 pairs of fluorine-substituted and nonfluorine-substituted compounds. We developed a multimodal deep learning model (F-CPI). In comparison with traditional machine learning and popular CPI task models, the accuracy, precision, and recall of F-CPI (∼90, ∼79, and ∼45%) were higher than those of GraphDTA (∼86, ∼58, and ∼40%). The application of the F-CPI for the structural optimization of hit compounds against SARS-CoV-2 3CLpro by F-substitution achieved a more than 100-fold increase in bioactivity (IC50: 0.23 μM vs 28.19 μM). Therefore, the multimodal deep learning model F-CPI would be a veritable and effective tool in the context of drug discovery and design.

Parkinson’s is the second most common neurodegenerative disease, affecting nearly 8.5M people and steadily increasing. In this research, Multimodal Deep Learning is investigated for the Prodromal stage detection of Parkinson's Disease (PD), combining different 3D architectures with the novel Excitation Network (EN) and supported by Explainable Artificial Intelligence (XAI) techniques. Utilizing data from the Parkinson’s Progression Markers Initiative, this study introduces a joint co-learning approach for multimodal fusion, enabling end-to-end training of deep neural networks and facilitating the learning of complementary information from both imaging and clinical modalities. DenseNet with EN outperformed other models, showing a substantial increase in accuracy when supplemented with clinical data. XAI methods, such as Integrated Gradients for ResNet and DenseNet, and Attention Heatmaps for Vision Transformer (ViT), revealed that DenseNet focused on brain regions believed to be critical to prodromal pathophysiology, including the right temporal and left pre-frontal areas. Similarly, ViT highlighted the lateral ventricles associated with cognitive decline, indicating their potential in the Prodromal stage. These findings underscore the potential of these regions as early-stage PD biomarkers and showcase the proposed framework's efficacy in predicting subtypes of PD and aiding in early diagnosis, paving the way for innovative diagnostic tools and precision medicine.

To address the problems of the traditional human motion gesture tracking and recognition methods, such as poor tracking effect, low recognition accuracy, high frame loss rate, and long-time cost, a dynamic human motion gesture tracking and recognition algorithm using multimode deep learning was proposed. Firstly, the collected human motion images are repaired in the three-dimensional (3D) environment, and the multimodal 3D human motion model is reconstructed using the processed images. Secondly, according to the results of model reconstruction, the camera gesture and other parameters of the keyframe are used to construct the target tracking optimization function so as to achieve the purpose of accurate tracking of human motion. Finally, for multimodal human motion gesture learning, a convolutional neural network (CNN) is developed. The trained CNN is utilized to complete dynamic human motion recognition after convolutional and pooling calculations. The results demonstrate that the proposed algorithm is effective in tracking human motion gestures. The average recognition accuracy is 96%, the average frame loss rate is 8.8%, the time cost is low, and the proposed algorithm has a high F-measure and much lower power consumption than other algorithms, indicating that the proposed algorithm is effective.