• Home
  • Search
  • VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
  • Cite Icon16
  • https://doi.org/10.18653/v1/2021.conll-1.3Copy DOI Icon

VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering

  • Jan 1, 2021
  • Ekta Sood +4 more
Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

We present VQA-MHUG -a novel 49participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-ofthe-art VQA models: Modular Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show -for the first time -that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.

Loading PDF

Similar Papers
  • Conference Article
  • Citations15

Feature Enhancement in Attention for Visual Question Answering

  • Jul 01, 2018
  • Yuetan Lin +3
  • Book Chapter

Boosting Visual Question Answering Through Geometric Perception and Region Features

  • Sep 28, 2023
  • Frontiers in artificial intelligence and applications
  • Hong Yu +3
  • Research Article
  • Citations17

Accuracy vs. complexity: A trade-off in visual question answering models

  • Jun 12, 2021
  • Pattern Recognition
  • Moshiur Farazi +2
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • Conference Article
  • Citations388

In Defense of Grid Features for Visual Question Answering

  • Jun 01, 2020
  • Huaizu Jiang +4
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • Conference Article
  • Citations1

Dynamic Co-attention Network for Visual Question Answering

  • Nov 26, 2021
  • Doaa B Ebaid +2
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
  • Research Article
  • Citations1

Adversarial Sample Synthesis for Visual Question Answering

  • Nov 21, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Chuanhao Li +4
  • PDF
  • Research Article
  • Citations14

An effective spatial relational reasoning networks for visual question answering.

  • Nov 28, 2022
  • PLOS ONE
  • Xiang Shen +4
  • PDF
  • Conference Article
  • Citations132

Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?

  • Jan 01, 2016
  • Abhishek Das +4
  • PDF
  • Research Article
  • Citations442

Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

  • Oct 01, 2017
  • Computer Vision and Image Understanding
  • Abhishek Das +4
  • PDF
  • Research Article
  • Citations19

Advancing surgical VQA with scene graph knowledge

  • May 23, 2024
  • International Journal of Computer Assisted Radiology and Surgery
  • Kun Yuan +5
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Conference Article
  • Citations47

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

  • Jul 18, 2019
  • Yangyang Guo +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.