A Multimodal Deep Learning Approach to Predicting Systemic Diseases from Oral Conditions.

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

According to the World Health Organization (WHO) oral conditions may be determined by social, biological, behavioral, and psychosocial factors. The study assessed oral health status and its determinants associated with oral health conditions among adult residents in Vilnius, Lithuania. A total of 453 of 35–74-year-olds participated (response rate 63%). A self-reported questionnaire was administered. Dental caries experience (D3MFS score), periodontal probing depth (PPD), andnumber of missing teeth were assessed clinically. Data were analyzed using χ2 test, independent samples t-test, and multivariable linear regression. The mean (sd) of D3MFS scores was 67.3 (33.5), the mean (sd) number of teeth with PPD 4+ mm was 5.9 (5.3), prevalence of periodontitis was 33%, the mean (sd) number of missing teeth was 6.9 (6.8), and prevalence of total edentulism was 3.8%. Medication use was associated with all oral health conditions, while age was associated with caries experience, and missing teeth. Sugar-containing diet was associated with caries experience, and missing teeth, and smoking with caries experience and periodontal status. Systemic diseases were associated with periodontal status, while behavioral determinants, last dental visit, and use of fluoridated toothpaste were associated with missing teeth. Oral health status among adult Vilnius residents was poor. Oral conditions were associated with both biological and behavioral determinants. Oral health promotion should focus on modifying behavioral determinants.

Objective:Poor outcomes in acute respiratory distress syndrome (ARDS) can be alleviated with tools that support early diagnosis. Current machine learning methods for detecting ARDS do not take full advantage of the multimodality of ARDS pathophysiology. We developed a multimodal deep learning model that uses imaging data, continuously collected ventilation data, and tabular data derived from a patient’s electronic health record (EHR) to make ARDS predictions.Materials and Methods:A chest radiograph (x-ray), at least two hours of ventilator waveform (VWD) data within the first 24 hours of intubation, and EHR-derived tabular data were used from 220 patients admitted to the ICU to train a deep learning model. The model uses pretrained encoders for the x-rays and ventilation data and trains a feature extractor on tabular data. Encoded features for a patient are combined to make a single ARDS prediction. Ablation studies for each modality assessed their effect on the model’s predictive capability.Results:The trimodal model achieved an area under the receiver operator curve (AUROC) of 0.86 with a 95% confidence interval of 0.01. This was a statistically significant improvement (p<0.05) over single modality models and bimodal models trained on VWD+tabular and VWD+x-ray data.Discussion and Conclusion:Our results demonstrate the potential utility of using deep learning to address complex conditions with heterogeneous data. More work is needed to determine the additive effect of modalities on ARDS detection. Our framework can serve as a blueprint for building performant multimodal deep learning models for conditions with small, heterogeneous datasets.

ABSTRACTObjectivesTo investigate the link between self-reported oral health and arthritis in the Scottish population using data from the Scottish Health Survey.Material and MethodsData were available from 2008 to 2013 on self-reported arthritis, oral health conditions and oral hygiene habits from the Scottish Health Survey. Arthritis was defined in this survey by self-reported long standing illness, those who reported having arthritis, rheumatism and/or fibrositis. Oral conditions were defined by self-reported bleeding gums, toothache, biting difficulties and/or edentulousness. Oral hygiene habits were defined by self-reported brushing teeth and/or using dental floss on daily basis. Logistic regression was used for statistical analysis adjusted for age, gender, qualification, smoking and body mass index.ResultsPrevalence of self-reported arthritis was 9.3% (95% confidence interval [CI] = 9.03 to 9.57). Those who reported having bleeding gums (adjusted odds ratio [OR] = 1.63; 95% CI = 1.35 to 1.96), toothache (OR = 1.32; 95% CI = 1.16 to 1.5), biting difficulties (OR = 1.95; 95% CI = 1.62 to 2.34), and being edentulous (OR = 1.22; 95% CI = 1.08 to 1.37) had an increased risk of arthritis. Brushing teeth (OR = 1.25; 95% CI = 0.74 to 2.12), and using dental floss (OR = 1.11; 95% CI = 0.89 to 1.39) were not associated with arthritis.ConclusionsSelf-reported oral conditions were associated with increased risk of self-reported arthritis. Oral hygiene habits were not associated with self-reported arthritis. Further investigation is required to assess the causal association between oral hygiene, oral disease and arthritis.

This paper presents an approach for adaptive scheduling and robustness optimization in global logistics networks by integrating multimodal deep reinforcement learning with Internet of Things (IoT) technologies. We propose an integrated framework comprising a multimodal data fusion mechanism that synthesizes heterogeneous IoT sensor data, historical records, and contextual information; an adaptive deep reinforcement learning architecture that generates dynamic scheduling policies; and a multi-objective robust optimization method that balances operational efficiency with system resilience. The framework addresses key challenges in global logistics including demand volatility, transportation disruptions, and environmental uncertainties. Comprehensive experiments conducted on real-world logistics datasets demonstrate that our approach outperforms traditional methods with an 18.7% reduction in operational costs, 12.4% improvement in service levels, and significantly enhanced robustness under various disruption scenarios. The proposed method maintains 83% performance stability during complex disruptions compared to 51–72% for alternative approaches, while keeping computational requirements feasible for practical deployment. This research demonstrates potential contributions to AI-driven logistics operations management by showing improved supply chain performance through multimodal learning and robust optimization techniques.

In recent years, deep learning technology has made breakthroughs in computer vision. After using large-scale data training, the deep neural network represented by GAN is significantly better than previous technologies in image generation, including generating more reasonable, higher-definition, more complex, and more accurate images. With the continuous development of datasets, models, and applications, the fusion of different modal information, including fusion of natural language, semantic layouts, tags, edge maps, and other different modal information, to generate images has become a new demand and challenge. There are related reviews on image generation and multimodal deep learning. However, there has not been a review dedicated to multimodal deep-learning image generation to discuss the current status, existing problems, and challenges of this task. Therefore, this review proposes a survey on multimodal deep learning image generation. It aims to provide readers with an application scenario for multimodal deep learning image generation. Also, it provides readers with new multimodal deep learning image generation technologies, the relevant datasets, evaluation metrics used, and some results comparison. Finally, this article describes some of the challenges and future topics of multimodal deep learning image generation.

Degenerative meniscus tears are often accompanied by varying degrees of osteoarthritis, making the prognostic outcome of arthroscopic partial meniscectomy (APM) difficult to predict. Our research objective is to develop and validate a multimodal deep learning radiology (MDLR) model based on the integration of multimodal data using deep learning radiology (DLR) scores from preoperative magnetic resonance imaging (MRI) images and clinical variables. From February 2020 to February 2022, 452 eligible patients with degenerative meniscus tear who underwent APM were retrospectively enrolled in cohorts. DLR features were extracted from MRI of the patient's knee. Then, an MDLR model was used for the patient prognosis after arthroscopy. The MDLR model for prognostic risk stratification incorporated DLR signatures and clinical variable. The standalone DLR model performed poorly, with a micro average receiver operating characteristic (ROC) curve and macro average ROC line of 0.780 and 0.765 in the training set, 0.747 and 0.747 in the validation set, and 0.720 and 0.732 in the test set, respectively, for predicting postoperative outcomes in degenerative meniscus tears. Multivariate analysis identified gender, height, weight, duration of pain, ESR, and VAS as indicators of poor prognosis. After combining the above clinical features, the performance of the MDLR model has been significantly improved, with the best performance achieved under the Light Gradient Boosting Machine (GBM) algorithm. The micro average ROC curve and macro average ROC line of this model for predicting the postoperative effect of degenerative meniscus tear were 0.917 and 0.919 in the training set, 0.874 and 0.882 in the validation set, and 0.921 and 0.951 in the test set, respectively. With these variables, the MDLR model provides four levels of prognosis for arthroscopic partial meniscectomy: Poor, pain relief 0-25%, Average, pain relief 25-50%, Good, pain relief 50-75%, Excellent, pain relief 75-100%. A tool based on MDLR was developed to consider that the pain exacerbation time is an important prognosis factor for arthroscopic partial meniscectomy in degenerative meniscus tear patients. MDLR showed outstanding performance for the prognostic efficiency stratification of degenerative meniscus tear patients who underwent arthroscopic partial meniscectomy and may help physicians with therapeutic decision making and surveillance strategy selection in clinical practice.

BackgroundAccurate prediction of pathological complete response (pCR) to neoadjuvant chemotherapy has significant clinical utility in the management of breast cancer treatment. Although multimodal deep learning models have shown promise for predicting pCR from medical imaging and other clinical data, their adoption has been limited due to challenges with interpretability and generalizability across institutions.MethodsWe developed a multimodal deep learning model combining post contrast-enhanced whole-breast MRI at pre- and post-treatment timepoints with non-imaging clinical features. The model integrates 3D convolutional neural networks and self-attention to capture spatial and cross-modal interactions. We utilized two public multi-institutional datasets to perform internal and external validation of the model. For model training and validation, we used data from the I-SPY 2 trial (N = 660). For external validation, we used the I-SPY 1 dataset (N = 114).ResultsOf the 660 patients in I-SPY 2, 217 patients achieved pCR (32.88%). Of the 114 patients in I-SPY 1, 29 achieved pCR (25.44%). The attention-based multimodal model yielded the best predictive performance with an AUC of 0.73 ± 0.04 on the internal data and an AUC of 0.71 ± 0.02 on the external dataset. The MRI-only model (internal AUC = 0.68 ± 0.03, external AUC = 0.70 ± 0.04) and the non-MRI clinical features-only model (internal AUC = 0.66 ± 0.08, external AUC = 0.71 ± 0.03) trailed in performance, indicating the combination of both modalities is most effective.ConclusionWe present a robust and interpretable deep learning framework for pCR prediction in breast cancer patients undergoing NAC. By combining imaging and clinical data with attention-based fusion, the model achieves strong predictive performance and generalizes across institutions.

Oral health plays a pivotal role in general health, especially in older people. Oral diseases may affect the development of systemic conditions, such as diabetes mellitus, cardiovascular disease, stroke and hypertension. The most important oral health conditions that have been recorded in dental literature for older population include tooth loss, dental caries, periodontal diseases, xerostomia (dry mouth) and oral cancer. Edentulism influences social life, either causing aesthetic problems or affecting functional abilities, such as speaking, chewing and eating. Dental caries in older people is similar to that in people in their thirties. Socio-economic status and living area play a key role in the development of dental caries. In addition, the accumulation of several risk factors, such as plaque or systemic diseases, acts synergistically in the onset of periodontal disease in seniors. Furthermore, older people, mainly due to their medications, exhibit a reduced amount of saliva. Xerostomia causes difficulties in chewing, speaking and swallowing, and it has a substantial impact on older people's lives. The prevalence of oral cancer is 1-10 per 100,000 patients, and several factors (smoking, alcohol, education, economic status) play crucial role. Limited data exists today that evaluates oral health conditions of seniors in the Balkan countries. Aging and socio-economic status of seniors in the Balkans are significantly associated with oral health problems.

The Brazilian population presents an ever-increasing number of sedentary people with poor oral health issues, including children and adolescents. Both physical inactivity and poor oral health have shown associations with various systemic diseases. The aim of this study was to verify the association between physical inactivity and oral health conditions. Three hundred and fifty-three adolescents (mean age 13 and 17 years). An epidemiological cross-sectional study. Assessment of the presence of caries was made by using the decayed, missing, filled (DMF) index. Gingival bleeding and quality of oral hygiene were also examined to evaluate the oral conditions. The International Physical Activity Questionnaire (PAQ-C) was used to assess physical activity. The confidence parameter was set at p=0.05. The results of this study showed that 74.3% were sedentary adolescents, the great majority of whom were girls (86.4%, p=0.001). There was a positive association between sedentary lifestyle and tooth decay (p=0.001), dental plaque (p=0.009), and time spent watching television (p=0.001). This study concluded that there was a positive association between physical inactivity and poor oral health.

Oral health status is influenced by numerous oral diseases and conditions, including loss of teeth and supporting dental alveolar bone, xerostomia, loss of taste and smell, orofacial pain, oral movement disorders, and others. Other major factors include general health, socioeconomic status (SES), nutritional well-being, and dietary habits (1). Diseases of the oral cavity, both local and systemic, can have a significant impact on ability to consume an adequate diet and consequently maintain optimal nutrition status. The impact of tooth loss, edentulism, and removable prostheses on dietary habits, diet adequacy, masticatory function, olfaction, and gastrointestinal (GI) disorders has been documented (2–11). Inadequate intake of fruits, vegetables, and whole grains is common in edentulous individuals or those individuals with maxillary and mandibular complete dentures, resulting in an inadequate intake of dietary fiber and vitamins A and C (5,9,10). Disorders of taste and smell can affect appetite and salivary flow and compromise dietary intake. Orofacial pain, salivary disorders, and oral movement disorders can also have a negative impact on appetite and impair normal oral function and eating ability. The relationship between oral disease and nutrition is synergistic. Oral diseases, along with acute, chronic, and terminal systemic diseases with oral manifestations, impact functional ability to eat as well as nutritional well-being (11).KeywordsOral HealthTooth LossOral DiseaseMasticatory PerformanceMasticatory FunctionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

Real time forecasting of disasters needs to be advanced and easy because with increasing disasters their frequency and severity. Traditional prediction can only be made with traditional disaster prediction methods: numerical weather prediction (NWP) models and remote sensing techniques, which are computationally inefficient, data sparse and cannot adapt to dynamic environmental changes. In order to overcome these limitations, this research presents a Transformer Based Multimodal Deep Learning Model to combine the existing multiple data sources ranging from satellite imagery, IoT sensor networks, meteorological observations etc., to meteorological and social media analytics. The model employs a multimodal fusion strategy, enabling dynamic feature selection and seamless integration of heterogeneous data streams. In contrast to the conventional deep learning techniques, such as CNNs and LSTMs, the transformer based model has excellent ability towards long-range dependency, reducing the latency of light inference and better computational efficiency. The results are proven to be 94% accurate, 91% precise and has 40% reduction in inferencer latency in real time, which makes it suitable for disaster forecasting. The advancement of the multimodal deep learning methodologies presents this research as one which serves to contribute to the AI driven disaster resilience. We will also work on future work in the form of advanced transformer variants, more data integration, and explainable AI (XAI) techniques for model interpretability and scalability. Finding have implications for the transformative potential of AI in climate adaptation and serve as a robust foundation for the next generation early warning systems and climate adaptation disaster risk mitigation across multiple sectors.

As the fashion e-commerce markets rapidly develop, tens of thousands of products are registered daily on e-commerce platforms. Individual sellers register products after setting up a product category directly on a fashion e-commerce platform. However, many sellers fail to find a suitable category and mistakenly register their products under incorrect ones. Precise category matching is important for increasing sales through search optimization and accurate product exposure. However, manually correcting registered categories is time-consuming and costly for platform managers. To resolve this problem, this study proposes a methodology for fashion e-commerce product classification based on multi-modal deep learning and transfer learning. Through the proposed methodology, three challenges in classifying fashion e-commerce products are addressed. First, the issue of extremely biased e-commerce data is addressed through under-sampling. Second, multi-modal deep learning enables the model to simultaneously use input data in different formats, which helps mitigate the impact of noisy and low-quality e-commerce data by providing richer information.Finally, the high computational cost and long training times involved in training deep learning models with both image and text data are mitigated by leveraging transfer learning. In this study, three strategies for transfer learning to fine-tune the image and text modules are presented. In addition, five methods for fusing feature vectors extracted from a single modal into one and six strategies for fine-tuning multi-modal models are presented, featuring a total of 14 strategies. The study shows that multi-modal models outperform unimodal models based solely on text or image. It also suggests the optimal conditions for classifying e-commerce products, helping fashion e-commerce practitioners construct models tailored to their respective business environments more efficiently.

BackgroundPathological complete response (pCR) to neoadjuvant systemic therapy (NAST) is an established prognostic marker in breast cancer (BC). Multimodal deep learning (DL), integrating diverse data sources (radiology, pathology, omics, clinical), holds promise for improving pCR prediction accuracy. This systematic review synthesizes evidence on multimodal DL for pCR prediction and compares its performance against unimodal DL.MethodsFollowing PRISMA, we searched PubMed, Embase, and Web of Science (January 2015–April 2025) for studies applying DL to predict pCR in BC patients receiving NAST, using data from radiology, digital pathology (DP), multi-omics, and/or clinical records, and reporting AUC. Data on study design, DL architectures, and performance (AUC) were extracted. A narrative synthesis was conducted due to heterogeneity.ResultsFifty-one studies, mostly retrospective (90.2%, median cohort 281), were included. Magnetic resonance imaging and DP were common primary modalities. Multimodal approaches were used in 52.9% of studies, often combining imaging with clinical data. Convolutional neural networks were the dominant architecture (88.2%). Longitudinal imaging improved prediction over baseline-only (median AUC 0.91 vs. 0.82). Overall, the median AUC across studies was 0.88, with 35.3% achieving AUC ≥ 0.90. Multimodal models showed a modest but consistent improvement over unimodal approaches (median AUC 0.88 vs. 0.83). Omics and clinical text were rarely primary DL inputs.ConclusionDL models demonstrate promising accuracy for pCR prediction, especially when integrating multiple modalities and longitudinal imaging. However, significant methodological heterogeneity, reliance on retrospective data, and limited external validation hinder clinical translation. Future research should prioritize prospective validation, integration underutilized data (multi-omics, clinical), and explainable AI to advance DL predictors to the clinical setting.