Multimodal Deep Learning for Crop Yield Prediction

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

As the fashion e-commerce markets rapidly develop, tens of thousands of products are registered daily on e-commerce platforms. Individual sellers register products after setting up a product category directly on a fashion e-commerce platform. However, many sellers fail to find a suitable category and mistakenly register their products under incorrect ones. Precise category matching is important for increasing sales through search optimization and accurate product exposure. However, manually correcting registered categories is time-consuming and costly for platform managers. To resolve this problem, this study proposes a methodology for fashion e-commerce product classification based on multi-modal deep learning and transfer learning. Through the proposed methodology, three challenges in classifying fashion e-commerce products are addressed. First, the issue of extremely biased e-commerce data is addressed through under-sampling. Second, multi-modal deep learning enables the model to simultaneously use input data in different formats, which helps mitigate the impact of noisy and low-quality e-commerce data by providing richer information.Finally, the high computational cost and long training times involved in training deep learning models with both image and text data are mitigated by leveraging transfer learning. In this study, three strategies for transfer learning to fine-tune the image and text modules are presented. In addition, five methods for fusing feature vectors extracted from a single modal into one and six strategies for fine-tuning multi-modal models are presented, featuring a total of 14 strategies. The study shows that multi-modal models outperform unimodal models based solely on text or image. It also suggests the optimal conditions for classifying e-commerce products, helping fashion e-commerce practitioners construct models tailored to their respective business environments more efficiently.

Multimodal deep learning for predicting the choice of cut parameters in the milling process

This paper presents an approach for adaptive scheduling and robustness optimization in global logistics networks by integrating multimodal deep reinforcement learning with Internet of Things (IoT) technologies. We propose an integrated framework comprising a multimodal data fusion mechanism that synthesizes heterogeneous IoT sensor data, historical records, and contextual information; an adaptive deep reinforcement learning architecture that generates dynamic scheduling policies; and a multi-objective robust optimization method that balances operational efficiency with system resilience. The framework addresses key challenges in global logistics including demand volatility, transportation disruptions, and environmental uncertainties. Comprehensive experiments conducted on real-world logistics datasets demonstrate that our approach outperforms traditional methods with an 18.7% reduction in operational costs, 12.4% improvement in service levels, and significantly enhanced robustness under various disruption scenarios. The proposed method maintains 83% performance stability during complex disruptions compared to 51–72% for alternative approaches, while keeping computational requirements feasible for practical deployment. This research demonstrates potential contributions to AI-driven logistics operations management by showing improved supply chain performance through multimodal learning and robust optimization techniques.

BackgroundPathological complete response (pCR) to neoadjuvant systemic therapy (NAST) is an established prognostic marker in breast cancer (BC). Multimodal deep learning (DL), integrating diverse data sources (radiology, pathology, omics, clinical), holds promise for improving pCR prediction accuracy. This systematic review synthesizes evidence on multimodal DL for pCR prediction and compares its performance against unimodal DL.MethodsFollowing PRISMA, we searched PubMed, Embase, and Web of Science (January 2015–April 2025) for studies applying DL to predict pCR in BC patients receiving NAST, using data from radiology, digital pathology (DP), multi-omics, and/or clinical records, and reporting AUC. Data on study design, DL architectures, and performance (AUC) were extracted. A narrative synthesis was conducted due to heterogeneity.ResultsFifty-one studies, mostly retrospective (90.2%, median cohort 281), were included. Magnetic resonance imaging and DP were common primary modalities. Multimodal approaches were used in 52.9% of studies, often combining imaging with clinical data. Convolutional neural networks were the dominant architecture (88.2%). Longitudinal imaging improved prediction over baseline-only (median AUC 0.91 vs. 0.82). Overall, the median AUC across studies was 0.88, with 35.3% achieving AUC ≥ 0.90. Multimodal models showed a modest but consistent improvement over unimodal approaches (median AUC 0.88 vs. 0.83). Omics and clinical text were rarely primary DL inputs.ConclusionDL models demonstrate promising accuracy for pCR prediction, especially when integrating multiple modalities and longitudinal imaging. However, significant methodological heterogeneity, reliance on retrospective data, and limited external validation hinder clinical translation. Future research should prioritize prospective validation, integration underutilized data (multi-omics, clinical), and explainable AI to advance DL predictors to the clinical setting.

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

Real time forecasting of disasters needs to be advanced and easy because with increasing disasters their frequency and severity. Traditional prediction can only be made with traditional disaster prediction methods: numerical weather prediction (NWP) models and remote sensing techniques, which are computationally inefficient, data sparse and cannot adapt to dynamic environmental changes. In order to overcome these limitations, this research presents a Transformer Based Multimodal Deep Learning Model to combine the existing multiple data sources ranging from satellite imagery, IoT sensor networks, meteorological observations etc., to meteorological and social media analytics. The model employs a multimodal fusion strategy, enabling dynamic feature selection and seamless integration of heterogeneous data streams. In contrast to the conventional deep learning techniques, such as CNNs and LSTMs, the transformer based model has excellent ability towards long-range dependency, reducing the latency of light inference and better computational efficiency. The results are proven to be 94% accurate, 91% precise and has 40% reduction in inferencer latency in real time, which makes it suitable for disaster forecasting. The advancement of the multimodal deep learning methodologies presents this research as one which serves to contribute to the AI driven disaster resilience. We will also work on future work in the form of advanced transformer variants, more data integration, and explainable AI (XAI) techniques for model interpretability and scalability. Finding have implications for the transformative potential of AI in climate adaptation and serve as a robust foundation for the next generation early warning systems and climate adaptation disaster risk mitigation across multiple sectors.

This paper provides a comprehensive review of multimodal deep learning models that utilize conversational data to detect mental health disorders. In addition to discussing models based on the Transformer, such as BERT (Bidirectional Encoder Representations from Transformers), this paper addresses models that existed prior to the Transformer, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The paper covers the application of these models in the construction of multimodal deep learning systems to detect mental disorders. In addition, the difficulties encountered by multimodal deep learning systems are brought up. Furthermore, the paper proposes research directions for enhancing the performance and robustness of these models in mental health applications. By shedding light on the potential of multimodal deep learning in mental health care, this paper aims to foster further research and development in this critical domain.

Objectives: To propose a multimodal deep learning method for the classification of benign and malignant breast non-mass lesions (NMLs) using grayscale and color Doppler ultrasound and to compare the performance of multi-modality and single-modality breast ultrasound (BUS) models. Methods: This retrospective study collected 248 pathologically confirmed NMLs from 241 female patients comprising grayscale and color Doppler BUS images from March 2018 to November 2024. Three types of convolutional neural networks (CNNs), including ResNet50, ResNet18, and VGG16, were evaluated as single-modality (grayscale or color Doppler) models via five-fold cross-validations. The optimal model for each single-modality approach was chosen as the backbone network for multimodal deep learning. Features extracted from grayscale and color Doppler BUS images were then concatenated to predict the probabilities of benignity and malignancy. The diagnostic efficacy of the multi-modality BUS models was comparatively evaluated against single-modality counterparts. Results: The single-modality VGG16 models outperformed the other two CNN types for both grayscale and color Doppler BUS using five-fold cross-validations. Additionally, single-modality grayscale models outperformed single-modality color Doppler models. With a mean accuracy of 91.54%, sensitivity of 94.15%, specificity of 87.30%, F1 score of 0.93, and area under the receiver operating characteristic curve (AUC) of 0.96, the multimodal VGG16 models performed better than single-modality counterparts. Conclusions: VGG 16-based multimodal ultrasound deep learning showed excellent diagnostic efficacy in distinguishing between benign and malignant NMLs, indicating therapeutic potential to help radiologists assess NMLs.

Objective:Poor outcomes in acute respiratory distress syndrome (ARDS) can be alleviated with tools that support early diagnosis. Current machine learning methods for detecting ARDS do not take full advantage of the multimodality of ARDS pathophysiology. We developed a multimodal deep learning model that uses imaging data, continuously collected ventilation data, and tabular data derived from a patient’s electronic health record (EHR) to make ARDS predictions.Materials and Methods:A chest radiograph (x-ray), at least two hours of ventilator waveform (VWD) data within the first 24 hours of intubation, and EHR-derived tabular data were used from 220 patients admitted to the ICU to train a deep learning model. The model uses pretrained encoders for the x-rays and ventilation data and trains a feature extractor on tabular data. Encoded features for a patient are combined to make a single ARDS prediction. Ablation studies for each modality assessed their effect on the model’s predictive capability.Results:The trimodal model achieved an area under the receiver operator curve (AUROC) of 0.86 with a 95% confidence interval of 0.01. This was a statistically significant improvement (p<0.05) over single modality models and bimodal models trained on VWD+tabular and VWD+x-ray data.Discussion and Conclusion:Our results demonstrate the potential utility of using deep learning to address complex conditions with heterogeneous data. More work is needed to determine the additive effect of modalities on ARDS detection. Our framework can serve as a blueprint for building performant multimodal deep learning models for conditions with small, heterogeneous datasets.

BackgroundAccurate prediction of pathological complete response (pCR) to neoadjuvant chemotherapy has significant clinical utility in the management of breast cancer treatment. Although multimodal deep learning models have shown promise for predicting pCR from medical imaging and other clinical data, their adoption has been limited due to challenges with interpretability and generalizability across institutions.MethodsWe developed a multimodal deep learning model combining post contrast-enhanced whole-breast MRI at pre- and post-treatment timepoints with non-imaging clinical features. The model integrates 3D convolutional neural networks and self-attention to capture spatial and cross-modal interactions. We utilized two public multi-institutional datasets to perform internal and external validation of the model. For model training and validation, we used data from the I-SPY 2 trial (N = 660). For external validation, we used the I-SPY 1 dataset (N = 114).ResultsOf the 660 patients in I-SPY 2, 217 patients achieved pCR (32.88%). Of the 114 patients in I-SPY 1, 29 achieved pCR (25.44%). The attention-based multimodal model yielded the best predictive performance with an AUC of 0.73 ± 0.04 on the internal data and an AUC of 0.71 ± 0.02 on the external dataset. The MRI-only model (internal AUC = 0.68 ± 0.03, external AUC = 0.70 ± 0.04) and the non-MRI clinical features-only model (internal AUC = 0.66 ± 0.08, external AUC = 0.71 ± 0.03) trailed in performance, indicating the combination of both modalities is most effective.ConclusionWe present a robust and interpretable deep learning framework for pCR prediction in breast cancer patients undergoing NAC. By combining imaging and clinical data with attention-based fusion, the model achieves strong predictive performance and generalizes across institutions.

Study question Can a multimodal deep learning approach integrating time-lapse imaging, clinical parameters, and genomic data predict embryo ploidy status in assisted reproductive technology (ART)? Summary answer The multimodal deep learning model can effectively predict ploidy status of embryos using merely embryo imaging. What is known already Current embryo selection relies on subjective morphological assessment or invasive preimplantation genetic testing for aneuploidy (PGT-A). Although PGT-A can diagnose embryo ploidy status and is claimed to enhance pregnancy rates, it presents several limitations including invasive biopsy, high cost, and potentially inaccuracies in detecting mosaicism. Recent artificial intelligence (AI) models show promise but primarily focus on viability prediction rather than genome-wide chromosomal analysis. Despite advances in time-lapse imaging and AI, no validated model currently integrates multimodal data for non-invasive ploidy prediction. Furthermore, existing methods lack the capability to distinguish between meiotic and mitotic chromosomal abnormalities, crucial for understanding developmental potential. Study design, size, duration We retrospectively analyzed 1,400 embryos from PGT cycles (2022-2024) using a stratified random sampling approach. We used ResNet152 as backbone to develop the model, using a 70/20/10 split for training, validation, and testing. Multiple logistic regression and bootstrapping (1000 iterations) were performed for statistical validation. Model performance was assessed using ROCAUC analysis. Participants/materials, setting, methods We included embryos from patients undergoing PGT at our center, cultured in EmbryoScope® time-lapse incubators. Imaging frames were extracted at 2-hour intervals. Morphokinetic parameters (from fertilization to blastocyst formation) and imaging texture features, e.g., density were analyzed. Chromosomal status was determined via whole-genome sequencing of trophectoderm biopsies using haplarithmisis. We then applied class activation mapping (CAM) heatmaps to visualize pixel regions predictive of ploidy outcomes for model interpretation. Main results and the role of chance The multimodal model achieved a ploidy prediction ROCAUC of 0.72, significantly outperforming morphology-only model (ROCAUC: 0.67, p &amp;lt; 0.001). Texture analysis revealed significant correlations between density patterns and ploidy status (Spearman’s ρ = 0.68, p &amp;lt; 0.001). Texture feature scores were significantly different across embryo development process, such as key indicators at fertilization, day3, and day5. CAM heatmap analysis identified inner cell mass region as the strongest morphological predictor. Limitations, reasons for caution Our study has limitations due to its retrospective nature and the single-center design, which may restrict generalizability. Selection bias may occur as we only included embryos progressing to biopsy. External validation across different laboratories and diverse datasets is needed to confirm these findings. Wider implications of the findings Our method demonstrated the potential to reduce reliance on invasive PGT-A, mitigating procedural risks and costs. By enabling non-invasive euploid embryo selection, we aim to enhance the accessibility to precision reproductive care and refine transfer strategies for chromosomal abnormal embryos. Trial registration number No

In recent years, deep learning technology has made breakthroughs in computer vision. After using large-scale data training, the deep neural network represented by GAN is significantly better than previous technologies in image generation, including generating more reasonable, higher-definition, more complex, and more accurate images. With the continuous development of datasets, models, and applications, the fusion of different modal information, including fusion of natural language, semantic layouts, tags, edge maps, and other different modal information, to generate images has become a new demand and challenge. There are related reviews on image generation and multimodal deep learning. However, there has not been a review dedicated to multimodal deep-learning image generation to discuss the current status, existing problems, and challenges of this task. Therefore, this review proposes a survey on multimodal deep learning image generation. It aims to provide readers with an application scenario for multimodal deep learning image generation. Also, it provides readers with new multimodal deep learning image generation technologies, the relevant datasets, evaluation metrics used, and some results comparison. Finally, this article describes some of the challenges and future topics of multimodal deep learning image generation.

To develop and validate a multimodal deep learning model that predicts treatment responses to intravitreal anti-vascular endothelial growth factor (anti-VEGF) injections in patients with diabetic macular oedema (DMO) by combining optical coherence tomography images and clinical data. This study included 107 DMO patients who received three consecutive anti-VEGF treatments. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity. The model's predictions were compared with those of retinal specialists. Among 107 patients, 65 showed good response and 42 showed poor response to treatment. The multimodal model achieved an AUROC of 0.962 (95% CI, 0.945-0.979), accuracy of 0.953 (95% CI, 0.933-0.973), sensitivity of 0.969 (95% CI, 0.951-0.987), and specificity of 0.928 (95% CI, 0.903-0.953) in the internal validation. The model outperformed retinal specialists, who achieved accuracies ranging from 0.571 to 0.857. The multimodal deep learning model demonstrated high accuracy in predicting anti-VEGF treatment responses in DMO patients. This approach could enable more personalised treatment strategies and optimal resource utilisation in ophthalmological care. Further validation with larger, multicentre datasets is warranted to confirm its clinical utility.

The application area of unmanned aerial vehicles increases significantly recent years due to progress in hardware and algorithms for data acquisition and processing. Object detection and classification (recognition) in imagery acquired by unmanned aerial vehicle are the key tasks for many applications, and usually in practice an operator solves these tasks. Growing amount of data of different types and of different nature provides the possibility for deep machine learning which nowadays shows high level results for object detection and recognition. Two key problems are to be solved for applying deep learning for object recognition task when dealing with multi-spectral imagery: (a) availability of representative dataset for neural network training and testing and (b) effective way of multi-spectral data fusion during neural network training. The paper proposes the approaches for solving these problems. For creating a representative dataset synthetic infra-red images are generated using several real infra-red images and 3D model of a given object. An technique for realistic infra-red texturing based on accurate infra-red image exterior orientation and 3D model pose estimation is developed. It allows in automated mode to produce datasets of required volume for deep learning and automatically generate ground truth data for neural network training and testing. Two approaches for multi-spectral data fusion for object recognition are developed and evaluated: data level fusion and results level fusion. The results of the evaluation of both techniques on generated multi-spectral dataset are presented and discussed.