• Home
  • Search
  • KnowZRel: Common Sense Knowledge-Based Zero-Shot Relationship Retrieval for Generalized Scene Graph Generation
  • Cite Icon5
  • https://doi.org/10.1109/tai.2025.3544177Copy DOI Icon

KnowZRel: Common Sense Knowledge-Based Zero-Shot Relationship Retrieval for Generalized Scene Graph Generation

Show More
  • Abstract
  • Highlights & Summary
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

A scene graph is a key image representation in visual reasoning. The generalizability of scene graph generation (SGG) methods is crucial for reliable reasoning and real-world applicability. However, imbalanced training datasets limit this, underrepresenting meaningful visual relationships. Current SGG methods using external knowledge sources face limitations due to these imbalances or restricted relationship coverage, impacting their reasoning and generalization capabilities. We propose a novel neurosymbolic approach that integrates data-driven object detection with heterogeneous knowledge graph-based object refinement and zero-shot relationship retrieval, highlighting the loosely coupled synergy between neural and symbolic components. This combination addresses the limitations of imbalanced training datasets in SGG and enables effective prediction of unseen visual relationships. Objects are detected using a region-based deep neural network and refined based on their positional and structural similarity, followed by retrieval of pairwise visual relationships using a heterogeneous knowledge graph. The redundant and irrelevant visual relationships are discarded based on the similarity of relationship labels and node embeddings. Finally, the visual relationships are interlinked to generate the scene graph. The employed heterogeneous knowledge graph combines diverse knowledge sources, offering rich common sense knowledge about objects and their interactions in the world. Our method, evaluated using the benchmark visual genome (VG) dataset and zero-shot recall (zR@K) metric, shows a 59.96% improvement over existing state-of-the-art methods, highlighting its effectiveness in generalized SGG. The object refinement step effectively improved the object detection performance by 57.1%. Additional evaluation using the GQA dataset confirms the cross-dataset generalizability of our method. We also compared various knowledge sources and embedding models to determine an optimal combination for zero-shot SGG. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jaleedkhan/zsrr-sgg</uri>.

Similar Papers
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
  • Research Article
  • Citations7

NeuSyRE: Neuro-symbolic visual understanding and reasoning framework based on scene graph enrichment

  • Oct 04, 2024
  • Semantic Web
  • M Jaleed Khan +2
  • Book Chapter
  • Citations14

Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning

  • Jan 01, 2022
  • Muhammad Jaleed Khan +2
  • PDF
  • Research Article
  • Citations9

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

  • Mar 21, 2025
  • Neurosymbolic Artificial Intelligence
  • M Jaleed Khan +3
  • PDF
  • Research Article
  • Citations13

Common Sense Knowledge Infusion for Visual Understanding and Reasoning: Approaches, Challenges, and Applications

  • Jul 01, 2022
  • IEEE Internet Computing
  • Muhammad Jaleed Khan +2
  • Book Chapter

Neurosymbolic Visual Reasoning with Scene Graphs and Multimodal LLMs

  • Mar 17, 2025
  • Frontiers in artificial intelligence and applications
  • M Jaleed Khan +2
  • PDF
  • Research Article
  • Citations3

HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group-Activity Scene Graph Generation in Videos.

  • May 24, 2024
  • Sensors (Basel, Switzerland)
  • Naga Venkata Sai Raviteja Chappa +4
  • Research Article
  • Citations16

DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media Posts

  • Jul 01, 2023
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Ze Fu +6
  • Research Article

Balanced scene graph generation assisted by an additional biased predictor

  • Nov 01, 2022
  • SCIENTIA SINICA Informationis
  • 文彬 王 +2
  • PDF
  • Conference Article
  • Citations6

Consistent Scene Graph Generation by Constraint Optimization

  • Oct 10, 2022
  • Boqi Chen +4
  • Conference Article
  • Citations757

Unbiased Scene Graph Generation From Biased Training

  • Jun 01, 2020
  • Kaihua Tang +4
  • Conference Article
  • Citations11

Mask and Predict

  • Oct 17, 2021
  • Hongshuo Tian +6
  • Research Article
  • Citations18

Tackling the Challenges in Scene Graph Generation With Local-to-Global Interactions.

  • Dec 01, 2023
  • IEEE Transactions on Neural Networks and Learning Systems
  • Sangmin Woo +2
  • Research Article
  • Citations4

Adversarial Attacks on Scene Graph Generation

  • Jan 01, 2024
  • IEEE Transactions on Information Forensics and Security
  • Mengnan Zhao +4
  • Conference Article
  • Citations363

Scene Graph Generation With External Knowledge and Image Reconstruction

  • Jun 01, 2019
  • Jiuxiang Gu +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.