Leveraging Textual Anatomical Knowledge for Class-Imbalanced Semi-Supervised Multi-Organ Segmentation.

This study aimed to evaluate the diagnostic accuracy of multimodal large language models in classifying superior labial frenulum attachments from intraoral photographs using expert consensus as the reference standard. Five experts (two periodontists and three orthodontists) established the consensus standard by classifying frenulum attachments in 117 intraoral images as mucosal, gingival, papillary, and papilla penetrating. The same photographs were then presented to three multimodal large language models (ChatGPT 4o, Gemini 2.5 Pro, and Microsoft Copilot GPT-4), and their diagnostic performance was evaluated using accuracy, sensitivity, specificity, and F1 score. Reliability was assessed using Fleiss' and Cohen's Kappa, and diagnostic performances were compared using Cochran's Q test. Human raters demonstrated almost perfect agreement (κ = 0.838, p < 0.001), whereas large language models showed poor inter-model agreement (κ = -0.124, p < 0.001). ChatGPT achieved slight significant agreement with the consensus (κ = 0.114, p = 0.019), although its clinical relevance was negligible. Gemini (κ = 0.099) and Copilot (κ = 0.027) showed no significant agreement (p > 0.05). Copilot yielded the highest overall accuracy (46.2%), followed by Gemini (44.5%) and ChatGPT (35.0%). The performance of the large language models varied across frenulum types. Current multimodal large language models demonstrate inconsistent and clinically insufficient accuracy in the classification of superior labial frenulum attachments from photographs. Domain-specific training is essential before large language models can be considered reliable diagnostic tools in dentistry.

Pneumothorax requires rapid recognition and accurate interpretation of chest X-rays (CXRs), particularly in acute settings where delays can have serious consequences. With the emergence of advanced image interpretation models capable of visual analysis, their diagnostic reliability in radiology practice remains to be determined. This study aimed to assess the diagnostic performance of three state-of-the-art systems in detecting pneumothorax using a large, well-annotated dataset. A total of 10,675 CXRs from the publicly available SIIM-ACR Pneumothorax Segmentation dataset were analyzed. Three multimodal models (GPT-4o, Gemini 2 Pro, and Claude 4 Sonnet) were evaluated using a uniform, image-based approach. Each model's binary outputs (presence: 1, absence: 0) were compared with reference results to determine accuracy, sensitivity, specificity, precision, and F1 scores. Additional subgroup analyses were conducted across pneumothorax size categories: small, medium, and large. Pairwise statistical comparisons were performed using McNemar's test. Sensitivity, specificity, and overall accuracy are reported with corresponding 95% confidence intervals. The prevalence of pneumothorax in the dataset was 22.3% (n = 2,379). All models demonstrated high specificity (above 0.90) but consistently low sensitivity (0.16-0.36). The best overall performance was observed with Gemini 2, which achieved an accuracy of 0.79 and specificity of 0.95, whereas Claude 4 showed greater sensitivity (0.20-0.34) across lesion-size categories. Diagnostic performance improved with increasing pneumothorax size, but smaller lesions remained difficult to identify. Pairwise comparisons confirmed statistically significant differences among all evaluated systems (P < 0.050). In this large-scale evaluation, the tested models exhibited strong reliability in identifying normal examinations but limited ability to detect subtle or small pneumothoraxes. Despite high specificity, low sensitivity limits the use of current Multimodal large language models as rule-out tools for pneumothorax. With continued refinement, these models may eventually support radiologists by improving workflow efficiency and diagnostic confidence. Automated systems capable of high specificity but low sensitivity should not be relied upon to exclude pneumothorax. However, they may serve as valuable assistants for confirming positive findings and prioritizing urgent cases in busy clinical workflows.

Nuclei segmentation and classification provide an essential basis for tumor immune microenvironment analysis. The previous nuclei segmentation and classification models require splitting large images into smaller patches for training, leading to two significant issues. First, nuclei at the borders of adjacent patches often misalign during inference. Second, this patch-based approach significantly increases the model's training and inference time. Recently, Mamba has garnered attention for its ability to model large-scale images with linear time complexity and low memory consumption. It offers a promising solution for training nuclei segmentation and classification models on full-sized images. However, the Mamba orientation-based scanning method lacks account for category-specific features, resulting in suboptimal performance in scenarios with imbalanced class distributions. To address these challenges, this paper introduces a novel scanning strategy based on category probability sorting, which independently ranks and scans features for each category according to confidence from high to low. This approach enhances the feature representation of uncertain samples and mitigates the issues caused by imbalanced distributions. Extensive experiments conducted on four public datasets demonstrate that our method outperforms state-of-the-art approaches, delivering superior performance in nuclei segmentation and classification tasks.

The integration of visual encoders and large language models (LLMs) has driven recent progress in multimodal large language models (MLLMs). However, the scarcity of high-quality instruction-tuning data for vision-language tasks remains a challenge. The current leading paradigm, such as LLaVA, relies on language-only GPT-4 to generate data, which requires pre-annotated image captions and detection bounding boxes, suffering from understanding image details. A practical solution to this problem would be to utilize the available multimodal large language models to generate instruction data for vision-language tasks. However, it's worth noting that the currently accessible MLLMs are not as powerful as their LLM counterparts, as they tend to produce inadequate responses and generate false information. As a solution for addressing the current issue, this paper proposes the Visual Instruction Generation and Correction (VIGC) framework that enables multimodal large language models to generate instruction-tuning data and progressively enhance its quality on-the-fly. Specifically, Visual Instruction Generation (VIG) guides the vision-language model to generate diverse instruction-tuning data. To ensure generation quality, Visual Instruction Correction (VIC) adopts an iterative update mechanism to correct any inaccuracies in data produced by VIG, effectively reducing the risk of hallucination. Leveraging the diverse, high-quality data generated by VIGC, we finetune mainstream models and validate data quality based on various evaluations. Experimental results demonstrate that VIGC not only compensates for the shortcomings of language-only data generation methods, but also effectively enhances the benchmark performance. The models, datasets, and code are available at https://opendatalab.github.io/VIGC

Uncrewed aerial vehicles (UAVs) aerial remote sensing (RS) images offer higher spatial resolution compared with satellite RS images, providing more detailed scene information. The flexibility of UAV-mounted sensors allows for the simultaneous acquisition of multimodal data, including RGB images and depth maps. Depth information, serving as a supplementary cue, has been extensively utilized in semantic segmentation research. Nevertheless, the current investigations on red, green, blue-depth (RGB-D) image semantic segmentation predominantly concentrate on indoor or road scenarios. In UAV RS scenes, there exist numerous objects featuring similar visual texture manifestations. The introduction of depth geometric information can efficaciously enhance the semantic segmentation accuracy of complex ground objects. Concurrently, high-resolution RS images also present challenges to the modeling of global information in large-scale space. To address these challenges, we propose the first framework for semantic segmentation of RGB-D images in UAV RS, namely, SkyMamba. This framework effectively integrates visual texture information and geometric depth information. Simultaneously, it leverages the long-sequence modeling capability of the state-space model to efficiently capture global context information in the RS space, achieving efficient semantic segmentation of RS scenes. Furthermore, we introduce a novel benchmark dataset for semantic segmentation of UAV RS RGB-D images, named comprehensive urban-scale depth, which is divided into two versions based on coarse-grained and fine-grained semantic annotation strategies to provide diverse challenges. Experimental results demonstrate that the proposed method exhibits state-of-the-art performance on both versions of the dataset, particularly in handling imbalanced scene semantic class distributions. The code for this article is available at the following URL: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CapyLin-G/CUS-Depth</uri>

Sentiment analysis is crucial in understanding user opinions, emotions, and attitudes across various domains. Traditional sentiment analysis methods rely primarily on textual data, limiting their ability to capture the full context of human expression, often including multimodal elements such as images, audio, and videos. Existing approaches struggle with ambiguity, sarcasm, and lack of contextual awareness, reducing accuracy and effectiveness. To address these limitations, we propose a novel framework called Sentiment Analysis using Machine Learning with Multimodal Models (SA-ML-MM), which integrates text, images, and audio inputs using large language models (LLMs) enhanced with multimodal capabilities. Our approach leverages ML-based feature extraction and fusion techniques to improve sentiment classification accuracy. The proposed framework is applied in social media analysis, customer feedback interpretation, and emotion recognition. Experimental results demonstrate that SA-ML-MM significantly outperforms traditional text-based models, achieving higher accuracy and robustness in sentiment prediction. By incorporating multiple data modalities, our approach effectively captures nuanced emotional expressions, making sentiment analysis more precise and reliable.

In Multimodal Multiple-Choice Questions, distractors play a pivotal role in rigorously evaluating the cross-modal reasoning capabilities of Multimodal Large Language Models by serving as plausible yet incorrect options. A comprehensive and reliable evaluation of distractor quality is therefore imperative for fostering genuine progress in this domain. However, prevailing evaluation approaches face a fundamental dilemma: they either rely on model-based metrics that fail to fully capture semantic nuances, or depend on human evaluation, which is resource-intensive and prone to subjective bias. To address these limitations, we introduce a comprehensive suite of 9 automated metrics, spanning both intrinsic and extrinsic dimensions, to reliably quantify distractor quality. Leveraging this framework, we propose a metric-driven ensemble strategy for constructing hard benchmarks. Specifically, we aggregate candidate pools from diverse advanced baselines and rigorously select the optimal subset of distractors that yield the highest quality scores under our proposed metrics. Extensive evaluations involving 33 Multimodal Large Language Models across 16 diverse benchmarks demonstrate that our method generates distractors with significantly higher confusability, posing a more rigorous challenge to current state-of-the-art models.

Incorporating modality-specific intensity prior as text prompt for multimodal myocardial pathology segmentation.

Background and Purpose:Deep learning promises accurate target contouring for online adaptive MR-guided radiotherapy (MRgRT) in rectal cancer. However, delineating the mesorectal clinical target volume (CTV) remains challenging. Integrating planning-based contours, delineated offline before treatment, can provide anatomical shape and boundary information. This study evaluated deep learning-based segmentation and registration models to determine the optimal approach for incorporating planning contours into online rectal contouring.Materials and Methods:Deep learning-based segmentation and registration models, both U-Nets, were developed using MRI of 104 rectal cancer patients, split into 68, 14, and 22 training, validation, and testing subjects. The segmentation model used the planning CTV and daily fraction MRI, while the registration model used the planning MRI and CTV and the daily fraction MRI. The models were compared in terms of contour accuracy (maximum Hausdorff distance (HD), Dice, and a qualitative score) and robustness against domain shifts.Results:When incorporating the planning contour, the segmentation and registration models achieved comparable median HD values of 9.3 mm (interquartile range, IQR: 7.1-12.1) and 10.2 (8.2-12.4) (p=0.18), respectively. However, segmentation achieved lower HD values in the middle and cranial regions of the target (HDmiddle: 5.3 mm (4.3-6.6) vs. 6.0 mm (4.8-8.0), p0.05; HDcranial: 7.6 mm (6.3-10.7) vs. 9.6 mm (7.5-11.9), p0.05). In addition, segmentation resulted in more clinically acceptable contours (9/10 versus 3/10) and was more robust to rectum volume variations than registration.Conclusion:Deep learning-based segmentation was identified as the optimal approach for incorporating the planning CTV into online rectal delineation in MRgRT.

Large Vision-Language Models (LVLMs) have demonstrated promising capabilities in medical image captioning task. Their architecture usually integrates a vision encoder, a Large Language Model (LLM), and a connector specifically designed to bridge the modality gap between vision and language. However, a light-weight connector struggles to filter appropriate visual features for an LLM to generate structured captions for medical images across various modalities. This paper proposes MoE-MSC: Mixture of Experts (MoE) with Multi-Stream Connector (MSC) for modality-aware structured medical image captioning. The captions are structured with identified modality, anatomical structures, Region-of-Interest (RoI) analysis, lesion findings, and local-global relationships indicating potential impact of lesion findings in RoI to other regions. The MSC with multiple Cross Attentions projects visual features from a vision encoder to a representation that enables an LLM to generate captions focusing on different aspects in a caption related to modality and anatomical structures, RoI, and local-global relationships. Furthermore, the MoE enables modality-aware captioning, in which modality-specific visual features from a vision encoder are routed to specialized experts, offering scalability, enhanced interpretability, and reduced computational cost during inference. Our extensive experiments demonstrate the capabilities of our model with superior quantitative and qualitative results, compared to relevant methods. The source code is available at https://github.com/alshahriarrubel/MoE-MSC.

Building a three-dimensional model of early-stage zebrafish embryo brain

Behavior analysis across species represents a fundamental challenge in neuroscience, psychology, and ethology, typically requiring extensive expert knowledge and labor-intensive processes that limit research scalability and accessibility. We introduce BehaveAgent, an autonomous multimodal AI agent designed to automate behavior analysis from video input without retraining or manual intervention. Unlike conventional methods that require manual behavior annotation, video segmentation, task-specific model training, BehaveAgent leverages the reasoning capabilities of multimodal large language models (LLM) to generalize across novel behavioral domains without need for additional training. It integrates LLMs, vision-language models (VLMs), and large-scale visual grounding modules, orchestrated through a multimodal context memory and goal-directed attention mechanism, to enable robust zero-shot visual reasoning across species and experimental paradigms, including plants, insects, rodents, primates, and humans. Upon receiving a video input, BehaveAgent autonomously identifies the correct analysis strategy and performs end-to-end behavior analysis and interpretation without human supervision. Leveraging vision-language representations, it performs general-purpose tracking, pose estimation and segmentation. We demonstrate BehaveAgent’s universal applicability to autonomously (1) identify the behavioral paradigm and develop an action plan specialized for the identified paradigm, (2) identify relevant subjects and objects, (3) track those features, (4) identify behavioral sequences with explicit reasoning, (5) generate and execute code for targeted analysis and (6) generate comprehensive research reports that integrate behavioral findings with relevant scientific literature. Through interpretable agentic reasoning, BehaveAgent makes its internal decision-making process transparent, clarifying why particular features are tracked or behaviors inferred. By reducing the time and expertise required for behavior analysis, BehaveAgent introduces a scalable, generalizable, and explainable paradigm for advancing biological and behavioral research.

Multimodal large language models (MLLMs) capable of integrating visual and textual information represent a promising advancement for clinical applications requiring image interpretation. Wound care assessment, which demands simultaneous analysis of wound photographs and clinical data, provides an ideal domain to evaluate multimodal vs unimodal artificial intelligence capabilities against human expertise. This study aims to compare the performance of MLLMs, unimodal ChatGPT 5.0, and human clinical experts on a standardized wound care certification examination. This cross-sectional comparative study evaluated 3 participant groups on a 25-question wound care certification examination spanning 4 clinical domains (Diagnosis, Treatment, Complication Management, and Wound Subtype Knowledge). Participants included 3 MLLMs (Med-PaLM 2, LLaVA-Med, and BioGPT), 1 unimodal large language model (ChatGPT 5.0), and 4 human clinical experts (general surgeon, wound care nurse, and 2 internal medicine physicians). Statistical analyses included one-way ANOVA with Tukey post hoc tests and domain-specific Kruskal-Wallis comparisons. Human experts achieved the highest accuracy (mean 86%, SD 9.1%), followed by MLLMs (mean 78.7%, SD 12.2%), while ChatGPT 5.0 achieved 64% accuracy, failing the 70% certification threshold. Significant overall group differences were observed (F2,5=8.42, P=.02, η²=0.74). MLLMs significantly outperformed ChatGPT 5.0 (difference=14.7 percentage points, P=.03, Cohen d=1.38), with the multimodal advantage most pronounced in visually dependent domains: Diagnosis (81% vs 43%, P=.008) and Complication Management (72% vs 50%, P=.03). No multimodal advantage was observed for text-based Wound Subtype Knowledge (both 67%). Med-PaLM 2 achieved 92% accuracy, matching that of the wound care nurse, while the general surgeon achieved the highest overall performance (96%). MLLMs demonstrate significant performance advantages over unimodal artificial intelligence in wound care assessment, particularly for visually dependent clinical tasks. While human experts with specialized wound care experience maintain overall superiority, the point estimate of the top-performing MLLM (Med-PaLM 2, 92%) fell within the observed range of human scores; however, the underpowered comparison (power=0.52) and wide CIs preclude definitive conclusions regarding noninferiority or equivalence to human experts. These findings support the potential role of MLLMs as clinical decision-support tools, warranting further adequately powered validation studies.

Security concerns related to Large Language Models (LLMs) have been extensively explored; however, the safety implications for Multimodal Large Language Models (MLLMs), particularly in medical contexts (MedMLLMs), remain inadequately addressed. This paper investigates the security vulnerabilities of MedMLLMs, focusing on their deployment in clinical environments where the accuracy and relevance of question-and-answer interactions are crucial for addressing complex medical challenges. We introduce and redefine two attack types: mismatched malicious attack (2M-attack) and optimized mismatched malicious attack (O2M-attack), by integrating existing clinical data with atypical natural phenomena. Using the comprehensive 3MAD dataset that we developed, which spans a diverse range of medical imaging modalities and adverse medical scenarios, we performed an in-depth analysis and proposed the MCM optimization method. This approach significantly improves the attack success rate against MedMLLMs. Our evaluations, which include white-box attacks on LLaVA-Med and transfer (black-box) attacks on four other SOTA models, reveal that even MedMLLMs designed with advanced security mechanisms remain vulnerable to breaches. This study highlights the critical need for robust security measures to enhance the safety and reliability of open-source MedMLLMs, especially in light of the potential impact of jailbreak attacks and other malicious exploits in clinical applications. Warning: Medical jailbreaking may generate content that includes unverified diagnoses and treatment recommendations. Always consult professional medical advice.

Accurate prediction of communication link quality metrics is essential for vehicle-to-infrastructure (V2I) systems, enabling smooth handovers, efficient beam management, and reliable low-latency communication. The increasing availability of sensor data from modern vehicles motivates the use of multimodal large language models (MLLMs) because of their adaptability across tasks and reasoning capabilities. However, MLLMs inherently lack three-dimensional spatial understanding. To overcome this limitation, a lightweight, plug-and-play bird’s-eye view (BEV) injection connector is proposed. In this framework, a BEV of the environment is constructed by collecting sensing data from neighboring vehicles. This BEV representation is then fused with the ego vehicle’s input to provide spatial context for the large language model. To support realistic multimodal learning, a co-simulation environment combining CARLA simulator and MATLAB-based ray tracing is developed to generate RGB, LiDAR, GPS, and wireless signal data across varied scenarios. Instructions and ground-truth responses are programmatically extracted from the ray-tracing outputs. Extensive experiments are conducted across three V2I link prediction tasks: line-of-sight (LoS) versus non-line-of-sight (NLoS) classification, link availability, and blockage prediction. Simulation results show that the proposed BEV injection framework consistently improved performance across all tasks. The results indicate that, compared to an ego-only baseline, the proposed approach improves the macro-average of the accuracy metrics by up to 13.9%. The results also show that this performance gain increases by up to 32.7% under challenging rainy and nighttime conditions, confirming the robustness of the framework in adverse settings.