Multimodal Llms For Image-Based Fabric Composition Identification

Background Clinicians leverage information from multiple data sources, such as the electrocardiogram (ECG) and echocardiogram to diagnose and manage cardiovascular disease. Deep learning models have shown immense potential to enhance these clinical processes by capturing digital biomarkers of cardiovascular disease from diagnostic modalities. However, the development of multimodal algorithms capable of making joint inferences from multiple modalities, mirroring the clinicians’ decision-making, has remained an unmet need. Purpose We aimed to develop a multimodal foundation model of ECG and echocardiogram as an integrative deep learning approach to evaluating cardiovascular conditions. Methods Twelve-lead ECGs and transthoracic echocardiograms performed at a large academic health system between 2001-2021 were collected. Individuals with paired ECGs and echocardiograms recorded within 3 years were identified and the data was split at the individual level into training, validation, and test sets (80/10/10%). Using paired ECGs and echocardiograms, we developed a deep learning model in a self-supervised setting with contrastive loss and merged embeddings in a joint latent space (Figure 1). We evaluated the performance of this foundation model after fine-tuning for left ventricular systolic dysfunction (LVSD defined as left ventricular ejection fraction of 40% or less), atrial fibrillation, age, and sex as auxiliary tasks. The performance of the multimodal model was compared with those of supervised unimodal ECG-only and echocardiography-only models for each task. Results In total, 4,391,610 videos from 129,424 echocardiography studies paired with 1,094,656 ECGs from 53,876 unique individuals were included (67 ± 14 years of age, 43% female, Figure 1). The multimodal algorithm demonstrated excellent discrimination for disease classification in the test set. In classification of LVSD, for example, it achieved an area under the receiver operating characteristic (AUROC) curve of 0.94, area under precision recall curve (AUPRC) of 0.99 among individuals with LV ejection fraction greater than 40%, and AUPRC of 0.72 among those with LV ejection fraction of 40% or less. The multimodal model consistently outperformed the unimodal algorithms, including the ECG-only and echocardiography-only models in classification of LVSD (AUROC ECG+Echo 0.94, ECG 0.89, Echo 0.89), atrial fibrillation (AUROC ECG+Echo 0.95, ECG, 0.94, Echo 0.75), sex (AUROC ECG+Echo 0.93, ECG 0.87, Echo 0.78) and age (R for variance ECG+Echo 0.75, ECG 0.67, Echo 0.58, Figure 2). Conclusion We have developed a multimodal foundation model for joint inference from ECGs and echocardiograms for diagnosis of cardiovascular conditions. This novel approach outperforms the conventional unimodal models in disease diagnosis and represents an advance in integrating the clinical applications of artificial intelligence.Figure 1 Figure 2

To compare the diagnostic efficiencies of deep learning single-modal and multi-modal for the classification of benign and malignant breast mass lesions. We retrospectively collected data from 203 patients (207 lesions, 101 benign and 106 malignant) with breast tumors who underwent breast magnetic resonance imaging (MRI) before surgery or biopsy between January 2014 and October 2020. Mass segmentation was performed based on the three dimensions-region of interest (3D-ROI) minimum bounding cube at the edge of the lesion. We established single-modal models based on a convolutional neural network (CNN) including T2WI and non-fs T1WI, the dynamic contrast-enhanced (DCE-MRI) first phase was pre-contrast T1WI (d1), and Phases 2, 4, and 6 were post-contrast T1WI (d2, d4, d6); and Multi-modal fusion models with a Sobel operator (four_mods:T2WI, non-fs-T1WI, d1, d2). Training set (n=145), validation set (n=22), and test set (n=40). Five-fold cross validation was performed. Accuracy, sensitivity, specificity, negative predictive value, positive predictive value, and area under the ROC curve (AUC) were used as evaluation indicators. Delong's test compared the diagnostic performance of the multi-modal and single-modal models. All models showed good performance, and the AUC values were all greater than 0.750. Among the single-modal models, T2WI, non-fs-T1WI, d1, and d2 had specificities of 77.1%, 77.2%, 80.2%, and 78.2%, respectively. d2 had the highest accuracy of 78.5% and showed the best diagnostic performance with an AUC of 0.827. The multi-modal model with the Sobel operator performed better than single-modal models, with an AUC of 0.887, sensitivity of 79.8%, specificity of 86.1%, and positive prediction value of 85.6%. Delong's test showed that the diagnostic performance of the multi-modal fusion models was higher than that of the six single-modal models (T2WI, non-fs-T1WI, d1, d2, d4, d6); the difference was statistically significant (p = 0.043, 0.017, 0.006, 0.017, 0.020, 0.004, all were greater than 0.05). Multi-modal fusion deep learning models with a Sobel operator had excellent diagnostic value in the classification of breast masses, and further increase the efficiency of diagnosis.

Glioblastoma is a highly aggressive brain tumor with limited survival that poses challenges in predicting patient outcomes. The Karnofsky Performance Status (KPS) score is a valuable tool for assessing patient functionality and contributes to the stratification of patients with poor prognoses. This study aimed to develop a 6-month postoperative KPS prediction model by combining clinical data with deep learning-based image features from pre- and postoperative MRI scans, offering enhanced personalized care for glioblastoma patients. Using 1,476 MRI datasets from the Brain Tumor Segmentation Challenge 2020 public database, we pretrained two variational autoencoders (VAEs). Imaging features from the latent spaces of the VAEs were used for KPS prediction. Neural network-based KPS prediction models were developed to predict scores below 70 at 6 months postoperatively. In this retrospective single-center analysis, we incorporated clinical parameters and pre- and postoperative MRI images from 150 newly diagnosed IDH wild-type glioblastoma, divided into training (100 patients) and test (50 patients) sets. In training set, the performance of these models was evaluated using the area under the curve (AUC), calculated through fivefold cross-validation repeated 10 times. The final evaluation of the developed models assessed in the test set. Among the 150 patients, 61 had 6-month postoperative KPS scores below 70 and 89 scored 70 or higher. We developed three models: a clinical-based model, an MRI-based model, and a multimodal model that incorporated both clinical parameters and MRI features. In the training set, the mean AUC was 0.785±0.051 for the multimodal model, which was significantly higher than the AUCs of the clinical-based model (0.716±0.059, P = 0.038) using only clinical parameters and the MRI-based model (0.651±0.028, P<0.001) using only MRI features. In the test set, the multimodal model achieved an AUC of 0.810, outperforming the clinical-based (0.670) and MRI-based (0.650) models. The integration of MRI features extracted from VAEs with clinical parameters in the multimodal model substantially enhanced KPS prediction performance. This approach has the potential to improve prognostic prediction, paving the way for more personalized and effective treatments for patients with glioblastoma.

BackgroundAccurate preoperative evaluation of cT1N0M0 papillary thyroid carcinoma (PTC) is essential for guiding appropriate treatment strategies. Although ultrasound is widely used for clinical staging, it has limitations in detecting lymph node metastasis (LNM) and capsular invasion (CI), which may lead to misclassification of high-risk patients. Such undetected risks pose safety concerns for those undergoing radiofrequency ablation. This study aimed to develop an artificial intelligence (AI)-assisted predictive model that integrates ultrasound radiomics and deep learning features to improve the identification of LNM and CI, thereby enhancing risk stratification and optimizing treatment strategies for cT1N0M0 PTC patients.MethodsA total of 203 PTC patients were divided into high-risk (CI or LNM) and low-risk groups, with 142 assigned to the training set and 61 to the internal test set. Regions of interest delineation was performed using ITK-Snap. Radiomic features were extracted with PyRadiomics, and embedding features were obtained through the Vision Transformer (ViT) model. Risk-related features were selected using least absolute shrinkage and selection operator (LASSO), variance thresholding, and recursive feature elimination (RFE). Single-modal and multimodal models were developed using feature-level and decision-level fusion. Feature importance was assessed using Shapley Additive exPlanations (SHAP). Model performance was evaluated using recall, accuracy, and area under curve (AUC).ResultsAmong 1,001 radiomics features, 47 were selected via LASSO and RFE, and 15 relevant features from 768 ViT features. In the internal test set, NeuralNet models based on radiomics and 2D deep learning achieved AUCs of 0.756 and 0.708, respectively, and 0.829 and 0.840 in the training set. The multimodal RandomForest model outperformed single-modality models, with an AUC of 0.763 in the test set and 0.992 in the training set. Decision-level fusion models, such as DLRad_LF_Avg and DLRad_LF_Max, improved the external test set AUC to 0.843. SHAP analysis identified key features linked to tumor heterogeneity.ConclusionThe multimodal AI model effectively predicts high-risk cT1N0M0 PTC, outperforming single-modality models and aiding clinical decision-making.

ObjectivesThis study aimed to develop a multimodal predictive model that integrates clinical data, radiomics, and three-dimensional deep learning to forecast acute respiratory distress syndrome in patients with acute pancreatitis.MethodsThis retrospective study analyzed data from 759 patients with acute pancreatitis treated at three hospitals. Radiomics features were extracted from three-dimensional computed tomography images, and a three-dimensional deep learning model was developed using convolutional networks. These components were combined with clinical data using the XGBoost algorithm to construct a multimodal model. The performance of the model was compared with that of single-modal models and traditional scoring systems (Modified Computed Tomography Severity Index, Ranson score, and Bedside Index for Severity in Acute Pancreatitis), using area under the curve as the primary metric. Model interpretability was enhanced using variable importance analysis, SHapley Additive exPlanations, local interpretable model-agnostic explanations, calibration plots, and decision curve analysis.ResultsThe multimodal model achieved area under the curve values of 0.872 (training set) and 0.876 (test set), outperforming traditional scores (Modified Computed Tomography Severity Index: 0.747 and 0.759; Ranson score: 0.575 and 0.568; and Bedside Index for Severity in Acute Pancreatitis: 0.748 and 0.757, respectively) and single-modal models (radiomics: 0.638 and 0.727 and deep learning: 0.756 and 0.727, respectively).ConclusionBy integrating clinical tabular data, radiomics, and deep learning features, the multimodal model can predict the risk of acute respiratory distress syndrome in patients with acute pancreatitis at an early stage.

BackgroundThere is a recognized need for additional approaches to improve the accuracy of extrathyroidal extension (ETE) diagnosis in papillary thyroid carcinoma (PTC) before surgery. Up to now, multimodal ultrasound has been widely applied in disease diagnosis. We investigated the value of radiomic features extracted from multimodal ultrasound in the preoperative prediction of ETE.MethodsWe retrospectively pathologically confirmed PTC lesions in 235 patients from January 2019 to April 2022 in our hospital, including 45 ETE lesions and 205 non-ETE lesions. MaZda software was employed to obtain radiomics parameters in multimodal sonography. The most valuable radiomics features were selected by the Fisher coefficient, mutual information, probability of classification error and average correlation coefficient methods (F + MI + PA) in combination with the least absolute shrinkage and selection operator (LASSO) method. Finally, the multimodal model was developed by incorporating the clinical records and radiomics features through fivefold cross-validation with a linear support vector machine algorithm. The predictive performance was evaluated by sensitivity, specificity, accuracy, F1 scores and the area under the receiver operating characteristic curve (AUC) in the training and test sets.ResultsA total of 5972 radiomics features were extracted from multimodal sonography, and the 13 most valuable radiomics features were selected from the training set using the F + MI + PA method combined with LASSO regression. The multimodal prediction model yielded AUCs of 0.911 (95% CI 0.866–0.957) and 0.716 (95% CI 0.522–0.910) in the cross-validation and test sets, respectively. The multimodal model and radiomics model showed good discrimination between ETE and non-ETE lesions.ConclusionRadiomics features based on multimodal ultrasonography could play a promising role in detecting ETE before surgery.

Patient satisfaction after one year of distal radius fracture fixation is influenced by various aspects such as the surgical approach, the patient's physical functioning, and psychological factors. Hence, a multimodal machine learning prediction model combining traditional rating scales and postoperative X-ray images of patients was developed to predict patient satisfaction one year after surgery for personalized clinical treatment. In this study, we reviewed 385 patients who underwent internal fixation with a palmar plate or external fixation bracket fixation in 2018-2020. After one year of postoperative follow-up, 169 patients completed the patient wrist evaluation (PRWE), EuroQol5D (EQ-5D), and forgotten joint score-12 (FJS-12) questionnaires and were subjected to X-ray capture. The region of interest (ROI) of postoperative X-rays was outlined using 3D Slicer, and the training and test sets were divided based on the satisfaction of the patients. Python was used to extract 848 image features, and random forest embedding was used to reduce feature dimensionality. Also, a machine learning model combining the patient's functional rating scale with the downscaled X-ray-related image features was built, followed by hyperparameter debugging using the grid search method during the modeling process. The stability of the Radiomics and Integrated models was first verified using the five-fold cross-validation method, and then receiver operating characteristic curves, calibration curves, and decision curve analysis were used to evaluate the performance of the model on the training and test sets. The feature dimensionality reduction yielded 16 imaging features. The accuracy of the two models was 0.831, 0.784 and 0.966, 0.804 on the training and test sets, respectively. The area under the curve (AUC) values for the Radiomics and Integrated model were 0.937, 0.673 and 0.997, 0.823 for the training and test sets, respectively. The calibration curves and decision curve analysis (DCA) of the Integrated model for the training and test sets had a more accurate prediction probability and clinical significance than the Radiomics model. A multimodal machine learning predictive model combining imaging and patient functional rating scales demonstrated optimal predictive performance for one-year postoperative satisfaction in patients with radial fractures, providing a basis for personalized postoperative patient management.

We developed a multimodal ultrasound (US) deep learning (DL) fusion model to automatically classify early fibrosis in patients with chronic kidney disease (CKD). This prospective study included patients with CKD who underwent continuous gray-scale US, superb microvascular imaging, and strain elastography from May to November 2022. According to the pathological tubular atrophy and interstitial fibrosis score, patients were divided into minimal and mild groups (affected area ≤10% and 11 − 25% of the total cortical volume, respectively). The dataset was divided into training (70%) and test (30%) sets. A DL model combining the features of the three US modes was developed to predict early fibrosis in patients with CKD. We compared these findings with the area under the receiver operating characteristic curve (AUC) of the clinical model by analyzing the receiver operating characteristic curve in the test set. The AUC of single-mode DL based on gray-scale US, superb microvascular imaging, and strain elastography was 0.682, 0.745, and 0.648, respectively, while that of the multimodal US DL model was 0.86. The accuracy, specificity, and sensitivity of the multimodal US DL model were 0.779, 0.767, and 0.796, respectively, and the negative and positive predictive values were 0.842 and 0.706, respectively. The AUC of the multimodal US DL model was significantly better than that of the single-mode DL and clinical models. The DL algorithm developed using multimodal US images can effectively predict early fibrosis in patients with CKD with significantly greater accuracy than single-mode DL or clinical models.

Artificial intelligence developers are increasingly building language models with warm and friendly personas that millions of people now use for advice, therapy and companionship1. Here we show how this can create a significant trade-off: optimizing language models for warmth can undermine their performance, especially when users express vulnerability. We conducted controlled experiments on five different language models, training them to produce warmer responses, then evaluating them on consequential tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing inaccurate factual information and offering incorrect medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed feelings of sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard tests, revealing systematic risks that standard testing practices may fail to detect. Our findings suggest that training artificial intelligence systems to be warm may come at a cost to accuracy, and that warmth and accuracy may not be independent by default. As these systems are deployed at an unprecedented scale and take on intimate roles in people's lives, this trade-off warrants attention from developers, policymakers and users alike.

Log-linear model combination is the standard approach in LVCSR to combine several knowledge sources, usually an acoustic and a language model.Instead of using a single scaling factor per knowledge source, we make the scaling factor wordand pronunciation-dependent.In this work, we combine three acoustic models, a pronunciation model, and a language model for a Mandarin BN/BC task.The achieved error rate reduction of 2% relative is small but consistent for two test sets.An analysis of the results shows that the major contribution comes from the improved interdependency of language and acoustic model.

Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the training data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architecture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to utilize monolingual data such as backtranslation, shallow fusion, and cold fusion.

Multimodal large language models (MLLMs) capable of integrating visual and textual information represent a promising advancement for clinical applications requiring image interpretation. Wound care assessment, which demands simultaneous analysis of wound photographs and clinical data, provides an ideal domain to evaluate multimodal vs unimodal artificial intelligence capabilities against human expertise. This study aims to compare the performance of MLLMs, unimodal ChatGPT 5.0, and human clinical experts on a standardized wound care certification examination. This cross-sectional comparative study evaluated 3 participant groups on a 25-question wound care certification examination spanning 4 clinical domains (Diagnosis, Treatment, Complication Management, and Wound Subtype Knowledge). Participants included 3 MLLMs (Med-PaLM 2, LLaVA-Med, and BioGPT), 1 unimodal large language model (ChatGPT 5.0), and 4 human clinical experts (general surgeon, wound care nurse, and 2 internal medicine physicians). Statistical analyses included one-way ANOVA with Tukey post hoc tests and domain-specific Kruskal-Wallis comparisons. Human experts achieved the highest accuracy (mean 86%, SD 9.1%), followed by MLLMs (mean 78.7%, SD 12.2%), while ChatGPT 5.0 achieved 64% accuracy, failing the 70% certification threshold. Significant overall group differences were observed (F2,5=8.42, P=.02, η²=0.74). MLLMs significantly outperformed ChatGPT 5.0 (difference=14.7 percentage points, P=.03, Cohen d=1.38), with the multimodal advantage most pronounced in visually dependent domains: Diagnosis (81% vs 43%, P=.008) and Complication Management (72% vs 50%, P=.03). No multimodal advantage was observed for text-based Wound Subtype Knowledge (both 67%). Med-PaLM 2 achieved 92% accuracy, matching that of the wound care nurse, while the general surgeon achieved the highest overall performance (96%). MLLMs demonstrate significant performance advantages over unimodal artificial intelligence in wound care assessment, particularly for visually dependent clinical tasks. While human experts with specialized wound care experience maintain overall superiority, the point estimate of the top-performing MLLM (Med-PaLM 2, 92%) fell within the observed range of human scores; however, the underpowered comparison (power=0.52) and wide CIs preclude definitive conclusions regarding noninferiority or equivalence to human experts. These findings support the potential role of MLLMs as clinical decision-support tools, warranting further adequately powered validation studies.

Pneumothorax requires rapid recognition and accurate interpretation of chest X-rays (CXRs), particularly in acute settings where delays can have serious consequences. With the emergence of advanced image interpretation models capable of visual analysis, their diagnostic reliability in radiology practice remains to be determined. This study aimed to assess the diagnostic performance of three state-of-the-art systems in detecting pneumothorax using a large, well-annotated dataset. A total of 10,675 CXRs from the publicly available SIIM-ACR Pneumothorax Segmentation dataset were analyzed. Three multimodal models (GPT-4o, Gemini 2 Pro, and Claude 4 Sonnet) were evaluated using a uniform, image-based approach. Each model's binary outputs (presence: 1, absence: 0) were compared with reference results to determine accuracy, sensitivity, specificity, precision, and F1 scores. Additional subgroup analyses were conducted across pneumothorax size categories: small, medium, and large. Pairwise statistical comparisons were performed using McNemar's test. Sensitivity, specificity, and overall accuracy are reported with corresponding 95% confidence intervals. The prevalence of pneumothorax in the dataset was 22.3% (n = 2,379). All models demonstrated high specificity (above 0.90) but consistently low sensitivity (0.16-0.36). The best overall performance was observed with Gemini 2, which achieved an accuracy of 0.79 and specificity of 0.95, whereas Claude 4 showed greater sensitivity (0.20-0.34) across lesion-size categories. Diagnostic performance improved with increasing pneumothorax size, but smaller lesions remained difficult to identify. Pairwise comparisons confirmed statistically significant differences among all evaluated systems (P < 0.050). In this large-scale evaluation, the tested models exhibited strong reliability in identifying normal examinations but limited ability to detect subtle or small pneumothoraxes. Despite high specificity, low sensitivity limits the use of current Multimodal large language models as rule-out tools for pneumothorax. With continued refinement, these models may eventually support radiologists by improving workflow efficiency and diagnostic confidence. Automated systems capable of high specificity but low sensitivity should not be relied upon to exclude pneumothorax. However, they may serve as valuable assistants for confirming positive findings and prioritizing urgent cases in busy clinical workflows.

Integrating outside knowledge for reasoning in visio-linguistic tasks such as visual question answering (VQA) is an open problem. Given that pretrained language models have been shown to include world knowledge, we propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. More specifically, we verbalize the image contents and allow language models to better leverage their implicit knowledge to solve knowledge-intensive tasks. Focusing on a visual question answering task which requires external knowledge (OK-VQA), our contributions are: (i) a text-only model that outperforms pretrained multimodal (image-text) models of comparable number of parameters; (ii) confirmation that our text-only method is specially effective for tasks requiring external knowledge, as it is less effective in standard a VQA task (VQA 2.0); and (iii) our method attains results in the state-of-the-art when increasing the size of the language model. We also significantly outperform current multimodal systems, even though augmented with external knowledge. Our qualitative analysis on OK-VQA reveals that automatic captions often fail to capture relevant information in the images, which seems to be balanced by the better inference ability of the text-only language models. Our work opens up possibilities to further improve inference in visio-linguistic tasks.

BACKGROUNDLymphovascular invasion (LVI) is an independent prognostic factor in rectal cancer, but its assessment relies on postoperative pathology. Radiomics-based analysis of multimodal magnetic resonance imaging (MRI) can provide noninvasive preoperative prediction of LVI status, supporting precision treatment decisions.AIMTo construct a machine learning model based on multimodal MRI radiomics features for noninvasive preoperative prediction of LVI status in rectal cancer, providing decision support for individualized clinical treatment.METHODSA total of 278 patients with pathologically confirmed rectal cancer after surgery were retrospectively included and divided into training set (222 cases) and test set (56 cases) at an 8:2 ratio. Three sequences were used for scanning: Fat-suppressed T2-weighted imaging, diffusion-weighted imaging, and T1-weighted contrast-enhanced imaging. PyRadiomics software was used to extract radiomics features, which were then screened through stability assessment, variance filtering, correlation analysis, univariate screening, and least absolute shrinkage and selection operator regression for key features. Single-modal models, multimodal radiomics model, clinical model, and clinical-radiomics combined model were constructed respectively. Model performance was evaluated using receiver operating characteristic curves.RESULTSAmong 278 patients, 121 (43.5%) were LVI-positive. Twenty-three key features were selected from initial 4200 features. Multivariate analysis showed that tumor diameter ≥ 4 cm, carcinoembryonic antigen ≥ 5 ng/mL, poor differentiation, T3-4 staging, N1-2 staging, and positive perineural invasion were independent predictors of LVI. In the test set, single-modal models achieved area under the curve (AUC) of 0.708-0.775, multimodal radiomics model achieved AUC of 0.835, clinical model achieved AUC of 0.782, and the combined model performed best (AUC = 0.867, sensitivity = 0.840, specificity = 0.806). Hosmer-Lemeshow test showed good calibration for all models (P > 0.05). Decision curve analysis demonstrated that the combined model had maximum net benefit within threshold probability range of 0.15-0.65.CONCLUSIONMachine learning models based on multimodal MRI radiomics features can effectively predict LVI status in rectal cancer, with the combined model showing optimal performance, providing a valuable quantitative tool for preoperative clinical assessment and individualized treatment decision-making.