- Research Article
- 10.1109/access.2025.3645220
NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs
- Jan 01, 2025
- IEEE Access
- M Jaleed Khan + 2 more +2
Incorporating multimodal features and heterogeneous common sense knowledge in scene representation and visual reasoning techniques is essential for accurate and intuitive Visual Question Answering (VQA). Knowledge-enhanced multimodal VQA offers transformative potential across computer vision applications, including accessibility, healthcare, security and education. Existing VQA techniques often neglect rich semantic and relational information about image content or rely on potentially biased knowledge sources with limited coverage. Our Neurosymbolic Visual Question Answering (NeSyVQA) framework addresses this by enriching scene graphs with rich background knowledge from a heterogeneous knowledge graph and employing the enriched scene graphs in an attention-based scene graph reasoning network. The framework employs a cascade of deep neural networks, including Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks, for object detection and predicate classification within Scene Graph Generation (SGG). The initial scene graph is semantically enriched for improved expressiveness using background knowledge and related facts extracted from a heterogeneous knowledge graph. The enriched scene graphs are then employed for downstream VQA using an attention-based scene graph reasoning network. Promising evaluation results were obtained for SGG and VQA tasks on the General Question Answering (GQA) and Visual Genome (VG) benchmark datasets using their standard evaluation metrics. The proposed NeSyVQA framework outperformed the existing state-of-the-art SGG and VQA techniques with over 13% higher relationship recall rates in SGG and a 4% higher accuracy on open-ended questions in VQA, which underscores the efficacy of leveraging multimodal features and heterogeneous knowledge for complex visual reasoning. The source code is available at https://bit.ly/nesy-vqa.
Read more