• Home
  • Search
  • NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs
  • https://doi.org/10.1109/access.2025.3645220Copy DOI Icon

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Incorporating multimodal features and heterogeneous common sense knowledge in scene representation and visual reasoning techniques is essential for accurate and intuitive Visual Question Answering (VQA). Knowledge-enhanced multimodal VQA offers transformative potential across computer vision applications, including accessibility, healthcare, security and education. Existing VQA techniques often neglect rich semantic and relational information about image content or rely on potentially biased knowledge sources with limited coverage. Our Neurosymbolic Visual Question Answering (NeSyVQA) framework addresses this by enriching scene graphs with rich background knowledge from a heterogeneous knowledge graph and employing the enriched scene graphs in an attention-based scene graph reasoning network. The framework employs a cascade of deep neural networks, including Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks, for object detection and predicate classification within Scene Graph Generation (SGG). The initial scene graph is semantically enriched for improved expressiveness using background knowledge and related facts extracted from a heterogeneous knowledge graph. The enriched scene graphs are then employed for downstream VQA using an attention-based scene graph reasoning network. Promising evaluation results were obtained for SGG and VQA tasks on the General Question Answering (GQA) and Visual Genome (VG) benchmark datasets using their standard evaluation metrics. The proposed NeSyVQA framework outperformed the existing state-of-the-art SGG and VQA techniques with over 13% higher relationship recall rates in SGG and a 4% higher accuracy on open-ended questions in VQA, which underscores the efficacy of leveraging multimodal features and heterogeneous knowledge for complex visual reasoning. The source code is available at https://bit.ly/nesy-vqa.

Similar Papers
  • Research Article
  • Citations5

KnowZRel: Common Sense Knowledge-Based Zero-Shot Relationship Retrieval for Generalized Scene Graph Generation

  • Dec 01, 2025
  • IEEE Transactions on Artificial Intelligence
  • M Jaleed Khan +2
  • Research Article
  • Citations7

NeuSyRE: Neuro-symbolic visual understanding and reasoning framework based on scene graph enrichment

  • Oct 04, 2024
  • Semantic Web
  • M Jaleed Khan +2
  • PDF
  • Research Article
  • Citations9

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

  • Mar 21, 2025
  • Neurosymbolic Artificial Intelligence
  • M Jaleed Khan +3
  • Book Chapter
  • Citations14

Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning

  • Jan 01, 2022
  • Muhammad Jaleed Khan +2
  • Conference Article
  • Citations22

Lightweight Visual Question Answering using Scene Graphs

  • Oct 26, 2021
  • Sai Vidyaranya Nuthalapati +6
  • Book Chapter

Neurosymbolic Visual Reasoning with Scene Graphs and Multimodal LLMs

  • Mar 17, 2025
  • Frontiers in artificial intelligence and applications
  • M Jaleed Khan +2
  • Research Article
  • Citations2

Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning

  • Jun 28, 2022
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Jin Chen +2
  • Conference Article
  • Citations2

EdgeNet for efficient scene graph classification

  • Jul 18, 2022
  • Vivek B.S +4
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article
  • Citations20

Generating Triples With Adversarial Networks for Scene Graph Construction

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Matthew Klawonn +1
  • PDF
  • Research Article
  • Citations13

Common Sense Knowledge Infusion for Visual Understanding and Reasoning: Approaches, Challenges, and Applications

  • Jul 01, 2022
  • IEEE Internet Computing
  • Muhammad Jaleed Khan +2
  • Research Article
  • Citations3

Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features

  • Dec 21, 2023
  • Computational Intelligence
  • Mayank Agrawal +2
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Research Article
  • Citations10

Question-aware dynamic scene graph of local semantic representation learning for visual question answering

  • Apr 25, 2023
  • Pattern Recognition Letters
  • Jinmeng Wu +5
  • Conference Article
  • Citations28

Visual Question Answering over Scene Graph

  • Sep 01, 2019
  • Soohyeong Lee +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.