• Home
  • Search
  • Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.
  • Open Access IconOpen Access
  • Cite Icon42
  • https://doi.org/10.1109/tnnls.2020.3045034Copy DOI Icon

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Though beneficial for encouraging the visual question answering (VQA) models to discover the underlying knowledge by exploiting the input-output correlation beyond image and text contexts, the existing knowledge VQA data sets are mostly annotated in a crowdsource way, e.g., collecting questions and external reasons from different users via the Internet. In addition to the challenge of knowledge reasoning, how to deal with the annotator bias also remains unsolved, which often leads to superficial overfitted correlations between questions and answers. To address this issue, we propose a novel data set named knowledge-routed visual question reasoning for VQA model evaluation. Considering that a desirable VQA model should correctly perceive the image context, understand the question, and incorporate its learned knowledge, our proposed data set aims to cut off the shortcut learning exploited by the current deep embedding models and push the research boundary of the knowledge-based visual question reasoning. Specifically, we generate the question-answer pair based on both the visual genome scene graph and an external knowledge base with controlled programs to disentangle the knowledge from other biases. The programs can select one or two triplets from the scene graph or knowledge base to push multistep reasoning, avoid answer ambiguity, and balance the answer distribution. In contrast to the existing VQA data sets, we further imply the following two major constraints on the programs to incorporate knowledge reasoning. First, multiple knowledge triplets can be related to the question, but only one knowledge relates to the image object. This can enforce the VQA model to correctly perceive the image instead of guessing the knowledge based on the given question solely. Second, all questions are based on different knowledge, but the candidate answers are the same for both the training and test sets. We make the testing knowledge unused during training to evaluate whether a model can understand question words and handle unseen combinations. Extensive experiments with various baselines and state-of-the-art VQA models are conducted to demonstrate that there still exists a big gap between the model with and without groundtruth supporting triplets when given the embedded knowledge base. This reveals the weakness of the current deep embedding models on the knowledge reasoning problem.

Similar Papers
  • PDF
  • Research Article
  • Citations19

Advancing surgical VQA with scene graph knowledge

  • May 23, 2024
  • International Journal of Computer Assisted Radiology and Surgery
  • Kun Yuan +5
  • Research Article
  • Citations1

Adversarial Sample Synthesis for Visual Question Answering

  • Nov 21, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Chuanhao Li +4
  • Conference Article
  • Citations90

Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects

  • Jan 01, 2019
  • Gabriel Grand +1
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
  • Conference Article
  • Citations7

Learning to Ask Informative Sub-Questions for Visual Question Answering

  • Jun 01, 2022
  • Kohei Uehara +2
  • Conference Article

Estimation of Visual Contents from Human Brain Signals via VQA Based on Brain-Specific Attention

  • Jun 04, 2023
  • Ryo Shichida +4
  • Conference Article
  • Citations3

NLP Meets Vision for Visual Interpretation - A Retrospective Insight and Future directions

  • May 20, 2021
  • Ahmed Jamshed +1
  • Research Article

Visual Question Answering Model Using Transformer-Based Multi-Head Cross Attention Network with Optimized Gold Rush Octave Convolution Network

  • Oct 28, 2025
  • International Journal of Image and Graphics
  • J Jinu Sophia +2
  • Video Transcripts

Question Agnostic Attention for Visual Question Answering

  • Dec 29, 2020
  • Underline Science Inc.
  • Moshiur Farazi
  • Conference Article
  • Citations10

Question-Agnostic Attention for Visual Question Answering

  • Jan 10, 2021
  • Moshiur Farazi +2
  • Research Article

Visual question answering-based image-finding generation for pulmonary nodules on chest CT from structured annotations.

  • Mar 27, 2026
  • International journal of computer assisted radiology and surgery
  • Maiko Nagao +5
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Conference Article
  • Citations47

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

  • Jul 18, 2019
  • Yangyang Guo +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.