• Home
  • Search
  • Enhanced Reasoning via Multimodal LLMs and Collaborative Inference
  • https://doi.org/10.1109/tmm.2025.3590940Copy DOI Icon

Enhanced Reasoning via Multimodal LLMs and Collaborative Inference

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Visual Question Answering (VQA) is a prevalent task that can facilitate the perception of the real world by the visually impaired. However, many VQA models tend to rely on superficial correlations in datasets for predictions rather than genuine reasoning, limiting their real-world applicability. While existing methods address this issue by incorporating debiasing strategies during training, they typically assume prior knowledge of out-of-distribution (OOD) test sets and then tailor debiasing strategies and select optimal models on the basis of the OOD samples. This reliance on OOD test data, however, is unrealistic in practical applications. To address this, some works introduce test-time adaptation techniques to mitigate dataset shifts during model deployment. Despite their potential, these methods risk catastrophic forgetting as they update models at test time without access to the ground-truth answers or the source data. An emerging solution involves leveraging the extensive knowledge embedded in Large Language Models (LLMs) to support reasoning tasks, yet their language-only input restricts flexibility in multimodal tasks. To bridge this gap, we propose leveraging the zero-shot capability of Multimodal Large Language Models (MLLMs). To optimise computational efficiency, we introduce a novel VQA Collaborative Inference framework (VQA-CI) that integrates MLLMs (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> BLIP-2 Flan T5) with VQA specialists (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> UpDn). This framework initially processes samples through VQA specialists and subsequently determines the necessity for re-evaluation with MLLMs based on predefined bias and reliability indicators. Experiments on the GQA-OOD and VQA-CP v2 datasets show that our VQA-CI achieves significant performance gains, with accuracy improvements of around 6% over state-of-the-art methods, underscoring the effectiveness of our VQA-CI.

Similar Papers
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Research Article
  • Citations2

LMT++: Adaptively Collaborating LLMs With Multi-Specialized Teachers for Continual VQA in Robotic Surgical Videos.

  • Jan 01, 2025
  • IEEE transactions on medical imaging
  • Yuyang Du +8
  • Research Article
  • Citations5

Co-LLaVA: Efficient Remote Sensing Visual Question Answering via Model Collaboration

  • Jan 29, 2025
  • Remote Sensing
  • Fan Liu +5
  • Conference Article

Grounded Multi-modal Conversation for Zero-shot Visual Question Answering

  • Oct 05, 2025
  • Mohammad Reza Zarei +2
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article
  • Citations12

DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark

  • Feb 19, 2025
  • Remote Sensing
  • Haodong Li +2
  • Research Article

Chain-of-Cognition Through Multi-Agent Collaboration Against Cross-Modal Cognitive Interference

  • Jan 01, 2026
  • IEEE Transactions on Audio, Speech and Language Processing
  • Liqi He +4
  • Research Article
  • Citations1

Asking Questions to Alleviate Object Hallucination in Large Vision-Language Models

  • Jan 01, 2025
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Chao Bi +4
  • Research Article
  • Citations41

Prompting large language model with context and pre-answer for knowledge-based VQA

  • Mar 06, 2024
  • Pattern Recognition
  • Zhongjian Hu +3
  • Research Article
  • Citations124

Detecting and Preventing Hallucinations in Large Vision Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Anisha Gunjal +2
  • Supplementary Content
  • Citations50

Multimodal Large Language Models in Medical Imaging: Current State and Future Directions

  • Aug 08, 2025
  • Korean Journal of Radiology
  • Yoojin Nam +14
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
  • Conference Article
  • Citations6

Visual Robustness Benchmark for Visual Question Answering (VQA)

  • Feb 26, 2025
  • Md Farhan Ishmam +5
  • Research Article
  • Citations9

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Ziyi Yin +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.