• Home
  • Search
  • Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?
  • Open Access IconOpen Access
  • Cite Icon132
  • https://doi.org/10.18653/v1/d16-1092Copy DOI Icon

Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?

  • Jan 1, 2016
  • Abhishek Das +4 more
Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.

Loading PDF

Similar Papers
  • PDF
  • Research Article
  • Citations442

Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

  • Oct 01, 2017
  • Computer Vision and Image Understanding
  • Abhishek Das +4
  • Research Article
  • Citations91

Exploring Human-Like Attention Supervision in Visual Question Answering

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Tingting Qiao +2
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • Conference Article
  • Citations47

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

  • Jul 18, 2019
  • Yangyang Guo +5
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
  • Research Article
  • Citations1

Adversarial Sample Synthesis for Visual Question Answering

  • Nov 21, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Chuanhao Li +4
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • PDF
  • Conference Article
  • Citations16

VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering

  • Jan 01, 2021
  • Ekta Sood +4
  • PDF
  • Research Article
  • Citations19

Advancing surgical VQA with scene graph knowledge

  • May 23, 2024
  • International Journal of Computer Assisted Radiology and Surgery
  • Kun Yuan +5
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Conference Article
  • Citations15

Feature Enhancement in Attention for Visual Question Answering

  • Jul 01, 2018
  • Yuetan Lin +3
  • Research Article

Visual Question Answering Model Using Transformer-Based Multi-Head Cross Attention Network with Optimized Gold Rush Octave Convolution Network

  • Oct 28, 2025
  • International Journal of Image and Graphics
  • J Jinu Sophia +2
  • Research Article
  • Citations17

Accuracy vs. complexity: A trade-off in visual question answering models

  • Jun 12, 2021
  • Pattern Recognition
  • Moshiur Farazi +2
  • Conference Article
  • Citations7

Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenario

  • Nov 17, 2023
  • Yimin Sun +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.