• Home
  • Search
  • Grounded Multi-modal Conversation for Zero-shot Visual Question Answering
  • https://doi.org/10.1109/smc58881.2025.11343454Copy DOI Icon

Grounded Multi-modal Conversation for Zero-shot Visual Question Answering

  • Oct 5, 2025
  • Mohammad Reza Zarei +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Zero-shot visual question answering (VQA) poses a formidable challenge at the intersection of computer vision and natural language processing. Traditionally, this problem has been tackled using end-to-end pre-trained vision-language models (VLMs). However, recent advancements in large language models (LLMs) demonstrate their exceptional reasoning and comprehension abilities, making them valuable assets in multi-modal tasks, including zero-shot VQA. LLMs have been previously integrated with VLMs to solve zero-shot VQA in a conversation-based approach. However, while the focus in VQA tasks is often on specific regions rather than the entire image, this aspect has been overlooked in previous approaches. Consequently, the overall performance of the framework relies on the ability of the pre-trained VLM to locate the region of interest that is relevant to the requested visual information within the entire image. To address this challenge, this paper proposes Grounded Multi-modal Conversation for Zero-shot Visual Question Answering (GMC-VQA), a region-based framework that leverages the complementary strengths of LLMs and VLMs in a conversation-based approach. We employ a grounding mechanism to refine visual focus according to the semantics of the question and foster collaborative interaction between VLM and LLM, effectively bridging the gap between visual and textual modalities and enhancing comprehension and response generation for visual queries. We evaluate GMC-VQA across three diverse VQA datasets, achieving substantial average improvements of 10.04% over end-to-end VLMs and 2.52% over the state-of-the-art VLM-LLM communication-based framework, respectively. Our code is publicly available at https://github.com/mrzarei5/GMC-VQA.

Similar Papers
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article
  • Citations9

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Ziyi Yin +7
  • Research Article
  • Citations5

Co-LLaVA: Efficient Remote Sensing Visual Question Answering via Model Collaboration

  • Jan 29, 2025
  • Remote Sensing
  • Fan Liu +5
  • Book Chapter
  • Citations3

Optimal Image Feature Ranking and Fusion for Visual Question Answering

  • Sep 09, 2020
  • Sruthy Manmadhan +1
  • Research Article

Analysis of Deep Learning Models on VQA(Toloka)Dataset

  • Oct 06, 2025
  • The Asian Bulletin of Big Data Management
  • Fariha Shoukat +3
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Research Article

Code-Driven LLM Agent for One-Shot Explanatory Visual Question Answering

  • Dec 16, 2025
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Zuyi Zhou +4
  • Research Article

KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs

  • Mar 03, 2026
  • PeerJ Computer Science
  • Wenhu Wang +2
  • Conference Article
  • Citations4

3DVQA: Visual Question Answering for 3D Environments

  • May 01, 2022
  • Yasaman Etesam +2
  • Conference Article
  • Citations4

Static Correlative Filter Based Convolutional Neural Network for Visual Question Answering

  • Jan 01, 2018
  • Lijun Chen +3
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
  • Conference Article

A Crowdsourcing Tool for Data Augmentation in Visual Question Answering Tasks

  • Oct 16, 2018
  • Ramon Silva +4
  • Research Article
  • Citations2

LMT++: Adaptively Collaborating LLMs With Multi-Specialized Teachers for Continual VQA in Robotic Surgical Videos.

  • Jan 01, 2025
  • IEEE transactions on medical imaging
  • Yuyang Du +8
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Research Article
  • Citations53

Visual Question Answering: A Tutorial

  • Nov 01, 2017
  • IEEE Signal Processing Magazine
  • Damien Teney +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.