• Home
  • Search
  • Multi-Granularity Feature Interaction and Multi-Region Selection Based Triplet Visual Question Answering
  • Cite Icon2
  • https://doi.org/10.1109/tbdata.2024.3453750Copy DOI Icon

Multi-Granularity Feature Interaction and Multi-Region Selection Based Triplet Visual Question Answering

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Accurately locating the question-related regions in one given image is crucial for visual question answering (VQA). The current approaches suffer two limitations: (1) Dividing one image into multiple regions may lose parts of semantic information and original relationships between regions; (2) Choosing only one or all image regions to predict the answer may correspondingly result in the insufficiency or redundancy of information. Therefore, how to effectively mine the relationship between image regions and choose the relevant image regions are vital. In this paper, we propose a novel <b>M</b>ulti-granularity feature interaction and <b>M</b>ulti-region selection-based triplet VQA model (M2TVQA). To tackle the first limitation, we propose the multi-granularity feature interaction strategy that adaptively supplements the global coarse-granularity features with the regional fine-granularity features. To overcome the second limitation, we design the Top-<inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> learning strategy to adaptively select <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> most relevant image regions to the question, even if the selected regions are far away in space. Such a strategy can select as many relevant image regions as possible and reduce introducing noise. Finally, we construct the multi-modality triplet to predict the answer of VQA. Extended experiments on two public outside knowledge datasets OK-VQA and KRVQA verify the effectiveness of the proposed model.

Similar Papers
  • Research Article
  • Citations74

Radial Graph Convolutional Network for Visual Question Generation.

  • Apr 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Xing Xu +4
  • Conference Article
  • Citations11

Grad-Cam Aware Supervised Attention for Visual Question Answering for Post-Disaster Damage Assessment

  • Oct 16, 2022
  • Argho Sarkar +1
  • PDF
  • Research Article
  • Citations13

Integrating deep learning for visual question answering in Agricultural Disease Diagnostics: Case Study of Wheat Rust

  • Nov 15, 2024
  • Scientific Reports
  • Akash Nanavaty +8
  • Research Article
  • Citations32

Word-to-region attention network for visual question answering

  • Aug 02, 2018
  • Multimedia Tools and Applications
  • Liang Peng +6
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • PDF
  • Research Article
  • Citations24

Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering

  • Nov 26, 2020
  • Sensors (Basel, Switzerland)
  • Zihan Guo +1
  • Conference Article
  • Citations15

Feature Enhancement in Attention for Visual Question Answering

  • Jul 01, 2018
  • Yuetan Lin +3
  • Research Article
  • Citations36

Answer Questions with Right Image Regions: A Visual Attention Regularization Approach

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Yibing Liu +6
  • Research Article
  • Citations13

Local relation network with multilevel attention for visual question answering

  • Jan 20, 2020
  • Journal of Visual Communication and Image Representation
  • Bo Sun +3
  • PDF
  • Research Article
  • Citations20

Visual question answering model for fruit tree disease decision-making based on multimodal deep learning

  • Jan 05, 2023
  • Frontiers in Plant Science
  • Yubin Lan +5
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • Research Article
  • Citations54

Head and eye movements and the role of memory limitations in a visual search paradigm

  • Jan 15, 2008
  • Journal of Vision
  • Gregor Hardiess +2
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.