• Home
  • Search
  • Answer Questions with Right Image Regions: A Visual Attention Regularization Approach
  • Open Access IconOpen Access
  • Cite Icon36
  • https://doi.org/10.1145/3498340Copy DOI Icon

Answer Questions with Right Image Regions: A Visual Attention Regularization Approach

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Visual attention in Visual Question Answering (VQA) targets at locating the right image regions regarding the answer prediction, offering a powerful technique to promote multi-modal understanding. However, recent studies have pointed out that the highlighted image regions from the visual attention are often irrelevant to the given question and answer, leading to model confusion for correct visual reasoning. To tackle this problem, existing methods mostly resort to aligning the visual attention weights with human attentions. Nevertheless, gathering such human data is laborious and expensive, making it burdensome to adapt well-developed models across datasets. To address this issue, in this article, we devise a novel visual attention regularization approach, namely, AttReg, for better visual grounding in VQA. Specifically, AttReg first identifies the image regions that are essential for question answering yet unexpectedly ignored (i.e., assigned with low attention weights) by the backbone model. And then a mask-guided learning scheme is leveraged to regularize the visual attention to focus more on these ignored key regions. The proposed method is very flexible and model-agnostic, which can be integrated into most visual attention-based VQA models and require no human attention supervision. Extensive experiments over three benchmark datasets, i.e., VQA-CP v2, VQA-CP v1, and VQA v2, have been conducted to evaluate the effectiveness of AttReg. As a by-product, when incorporating AttReg into the strong baseline LMH, our approach can achieve a new state-of-the-art accuracy of 60.00% with an absolute performance gain of 7.01% on the VQA-CP v2 benchmark dataset. In addition to the effectiveness validation, we recognize that the faithfulness of the visual attention in VQA has not been well explored in literature. In the light of this, we propose to empirically validate such property of visual attention and compare it with the prevalent gradient-based approaches.

Similar Papers
  • Research Article
  • Citations32

Word-to-region attention network for visual question answering

  • Aug 02, 2018
  • Multimedia Tools and Applications
  • Liang Peng +6
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Research Article
  • Citations91

Exploring Human-Like Attention Supervision in Visual Question Answering

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Tingting Qiao +2
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Research Article
  • Citations4

Multi-stage reasoning on introspecting and revising bias for visual question answering

  • Oct 08, 2024
  • ACM Transactions on the Web
  • L An-An +9
  • Research Article
  • Citations4

Self-supervised knowledge distillation in counterfactual learning for VQA

  • Nov 25, 2023
  • Pattern Recognition Letters
  • Yandong Bi +4
  • Research Article
  • Citations13

Local relation network with multilevel attention for visual question answering

  • Jan 20, 2020
  • Journal of Visual Communication and Image Representation
  • Bo Sun +3
  • Conference Article
  • Citations1

Dynamic Co-attention Network for Visual Question Answering

  • Nov 26, 2021
  • Doaa B Ebaid +2
  • Research Article
  • Citations78

Rich Visual Knowledge-Based Augmentation Network for Visual Question Answering

  • Sep 17, 2020
  • IEEE Transactions on Neural Networks and Learning Systems
  • Liyang Zhang +6
  • Research Article

Visual Question Answering Model Using Transformer-Based Multi-Head Cross Attention Network with Optimized Gold Rush Octave Convolution Network

  • Oct 28, 2025
  • International Journal of Image and Graphics
  • J Jinu Sophia +2
  • Research Article
  • Citations43

Multitask Learning for Visual Question Answering.

  • Mar 01, 2023
  • IEEE Transactions on Neural Networks and Learning Systems
  • Jie Ma +5
  • PDF
  • Research Article
  • Citations24

Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering

  • Nov 26, 2020
  • Sensors (Basel, Switzerland)
  • Zihan Guo +1
  • Conference Article
  • Citations2558

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Jul 01, 2017
  • Yash Goyal +4
  • Research Article
  • Citations8

Attacking VQA Systems via Adversarial Background Noise

  • Aug 01, 2020
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Akshay Chaturvedi +1
  • Conference Article
  • Citations4

Transformer-based Sparse Encoder and Answer Decoder for Visual Question Answering

  • Oct 21, 2022
  • Longkun Peng +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.