• Home
  • Search
  • Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning
  • Cite Icon14
  • https://doi.org/10.1007/978-3-031-06981-9_6Copy DOI Icon

Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning

  • Jan 1, 2022
  • Muhammad Jaleed Khan +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Scene graph generation aims to capture the semantic elements in images by modelling objects and their relationships in a structured manner, which are essential for visual understanding and reasoning tasks including image captioning, visual question answering, multimedia event processing, visual storytelling and image retrieval. The existing scene graph generation approaches provide limited performance and expressiveness for higher-level visual understanding and reasoning. This challenge can be mitigated by leveraging commonsense knowledge, such as related facts and background knowledge, about the semantic elements in scene graphs. In this paper, we propose the infusion of diverse commonsense knowledge about the semantic elements in scene graphs to generate rich and expressive scene graphs using a heterogeneous knowledge source that contains commonsense knowledge consolidated from seven different knowledge bases. The graph embeddings of the object nodes are used to leverage their structural patterns in the knowledge source to compute similarity metrics for graph refinement and enrichment. We performed experimental and comparative analysis on the benchmark Visual Genome dataset, in which the proposed method achieved a higher recall rate ( $$R@K = 29.89, 35.4, 39.12$$ for $$K = 20, 50, 100$$ ) as compared to the existing state-of-the-art technique ( $$R@K = 25.8, 33.3, 37.8$$ for $$K = 20, 50, 100$$ ). The qualitative results of the proposed method in a downstream task of image generation showed that more realistic images are generated using the commonsense knowledge-based scene graphs. These results depict the effectiveness of commonsense knowledge infusion in improving the performance and expressiveness of scene graph generation for visual understanding and reasoning tasks.

Similar Papers
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
  • Research Article
  • Citations7

NeuSyRE: Neuro-symbolic visual understanding and reasoning framework based on scene graph enrichment

  • Oct 04, 2024
  • Semantic Web
  • M Jaleed Khan +2
  • PDF
  • Research Article
  • Citations9

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

  • Mar 21, 2025
  • Neurosymbolic Artificial Intelligence
  • M Jaleed Khan +3
  • Research Article
  • Citations5

KnowZRel: Common Sense Knowledge-Based Zero-Shot Relationship Retrieval for Generalized Scene Graph Generation

  • Dec 01, 2025
  • IEEE Transactions on Artificial Intelligence
  • M Jaleed Khan +2
  • Conference Article
  • Citations22

Lightweight Visual Question Answering using Scene Graphs

  • Oct 26, 2021
  • Sai Vidyaranya Nuthalapati +6
  • Conference Article
  • Citations2

EdgeNet for efficient scene graph classification

  • Jul 18, 2022
  • Vivek B.S +4
  • Research Article
  • Citations2

Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning

  • Jun 28, 2022
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Jin Chen +2
  • Conference Article

Cross-modal Image Retrieval Based on Scene Graphs

  • Jul 26, 2024
  • Bochuan He +2
  • Book Chapter
  • Citations32

Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration

  • Jan 01, 2021
  • Xuan Kan +2
  • Conference Article

Finding Images by Dialoguing with Image

  • Oct 15, 2019
  • Lejian Ren +5
  • Book Chapter

Neurosymbolic Visual Reasoning with Scene Graphs and Multimodal LLMs

  • Mar 17, 2025
  • Frontiers in artificial intelligence and applications
  • M Jaleed Khan +2
  • Research Article
  • Citations20

Generating Triples With Adversarial Networks for Scene Graph Construction

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Matthew Klawonn +1
  • Research Article
  • Citations16

DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media Posts

  • Jul 01, 2023
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Ze Fu +6
  • Research Article

Balanced scene graph generation assisted by an additional biased predictor

  • Nov 01, 2022
  • SCIENTIA SINICA Informationis
  • 文彬 王 +2
  • Research Article

Review on scene graph generation methods

  • Aug 12, 2024
  • Multiagent and Grid Systems
  • Monesh S +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.