• Home
  • Search
  • Lightweight Visual Question Answering using Scene Graphs
  • Open Access IconOpen Access
  • Cite Icon22
  • https://doi.org/10.1145/3459637.3482218Copy DOI Icon

Lightweight Visual Question Answering using Scene Graphs

  • Oct 26, 2021
  • Sai Vidyaranya Nuthalapati +6 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Visual question answering (VQA) is a challenging problem in machine perception, which requires a deep joint understanding of both visual and textual data. Recent research has advanced the automatic generation of high-quality scene graphs from images, while powerful yet elegant models like graph neural networks (GNNs) have shown great power in reasoning over graph-structured data. In this work, we propose to bridge the gap between scene graph generation and VQA by leveraging GNNs. In particular, we design a new model called Conditional Enhanced Graph ATtention network (CE-GAT) to encode pairs of visual and semantic scene graphs with both node and edge features, which is seamlessly integrated with a textual question encoder to generate answers through question-graph conditioning. Moreover, to alleviate the training difficulties of CE-GAT towards VQA, we enforce more useful inductive biases in the scene graphs through novel question-guided graph enriching and pruning. Finally, we evaluate the framework on one of the largest available VQA datasets (namely, GQA) with ground-truth scene graphs, achieving the accuracy of 77.87%, compared with the state of the art (namely, the neural state machine (NSM)), which gives 63.17%. Notably, by leveraging existing scene graphs, our framework is much lighter compared with end-to-end VQA methods (e.g., about 95.3% less parameters than a typical NSM).

Similar Papers
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
  • Book Chapter

Multi-view Visual Question Answering Dataset for Real Environment Applications

  • Jan 01, 2020
  • Yue Qiu +3
  • Book Chapter
  • Citations14

Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning

  • Jan 01, 2022
  • Muhammad Jaleed Khan +2
  • Conference Article
  • Citations2

EdgeNet for efficient scene graph classification

  • Jul 18, 2022
  • Vivek B.S +4
  • Research Article
  • Citations20

Generating Triples With Adversarial Networks for Scene Graph Construction

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Matthew Klawonn +1
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • Research Article
  • Citations2

Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning

  • Jun 28, 2022
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Jin Chen +2
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Conference Article
  • Citations28

Visual Question Answering over Scene Graph

  • Sep 01, 2019
  • Soohyeong Lee +3
  • Book Chapter

Neurosymbolic Visual Reasoning with Scene Graphs and Multimodal LLMs

  • Mar 17, 2025
  • Frontiers in artificial intelligence and applications
  • M Jaleed Khan +2
  • Research Article

Incorporating Verb Semantic Information in Visual Question Answering Through Multitask Learning Paradigm

  • Jun 01, 2020
  • International Journal of Semantic Computing
  • Mehrdad Alizadeh +1
  • Conference Article
  • Citations603

VizWiz Grand Challenge: Answering Visual Questions from Blind People

  • Jun 01, 2018
  • Danna Gurari +7
  • Book Chapter
  • Citations4

Scene Graph Generation with Geometric Context

  • Jan 01, 2022
  • Vishal Kumar +2
  • Research Article
  • Citations10

Question-aware dynamic scene graph of local semantic representation learning for visual question answering

  • Apr 25, 2023
  • Pattern Recognition Letters
  • Jinmeng Wu +5
  • Research Article

Visual question answering-based image-finding generation for pulmonary nodules on chest CT from structured annotations.

  • Mar 27, 2026
  • International journal of computer assisted radiology and surgery
  • Maiko Nagao +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.