• Home
  • Search
  • Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering
  • Open Access IconOpen Access
  • Cite Icon9
  • https://doi.org/10.1145/3618301Copy DOI Icon

Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Knowledge-based visual question answering not only needs to answer the questions based on images but also incorporates external knowledge to study reasoning in the joint space of vision and language. To bridge the gap between visual content and semantic cues, it is important to capture the question-related and semantics-rich vision-language connections. Most existing solutions model simple intra-modality relation or represent cross-modality relation using a single vector, which makes it difficult to effectively model complex connections between visual features and question features. Thus, we propose a cross-modality multiple relations learning model, aiming to better enrich cross-modality representations and construct advanced multi-modality knowledge triplets. First, we design a simple yet effective method to generate multiple relations that represent the rich cross-modality relations. The various cross-modality relations link the textual question to the related visual objects. These multi-modality triplets efficiently align the visual objects and corresponding textual answers. Second, to encourage multiple relations to better align with different semantic relations, we further formulate a novel global-local loss. The global loss enables the visual objects and corresponding textual answers close to each other through cross-modality relations in the vision-language space, and the local loss better preserves semantic diversity among multiple relations. Experimental results on the Outside Knowledge VQA and Knowledge-Routed Visual Question Reasoning datasets demonstrate that our model outperforms the state-of-the-art methods.

Similar Papers
  • Conference Article

VIG: Visual Information-Guided Knowledge-Based Visual Question Answering

  • May 08, 2024
  • Heng Liu +5
  • Research Article
  • Citations1

ICQ-TransE: LLM-Enhanced Image-Caption-Question Translating Embeddings for Knowledge-Based Visual Question Answering

  • Jan 01, 2025
  • IEEE Transactions on Artificial Intelligence
  • Heng Liu +5
  • Research Article
  • Citations19

Answering knowledge-based visual questions via the exploration of Question Purpose

  • Aug 29, 2022
  • Pattern Recognition
  • Lingyun Song +5
  • Research Article
  • Citations4

Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference.

  • Dec 01, 2024
  • IEEE transactions on pattern analysis and machine intelligence
  • Kai Shen +6
  • Conference Article
  • Citations90

Multi-Modality Latent Interaction Network for Visual Question Answering

  • Oct 01, 2019
  • Gao Peng +4
  • Conference Article
  • Citations11

Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering

  • Jan 01, 2024
  • Junnan Dong +5
  • Conference Article
  • Citations3

Nested Attention Network with Graph Filtering for Visual Question and Answering

  • Jun 04, 2023
  • Jing Lu +4
  • PDF
  • Conference Article
  • Citations27

Improving Visual Question Answering by Referring to Generated Paragraph Captions

  • Jan 01, 2019
  • Hyounghun Kim +1
  • PDF
  • Research Article
  • Citations1

Learning to Reason on Tree Structures for Knowledge-Based Visual Question Answering

  • Feb 17, 2022
  • Sensors (Basel, Switzerland)
  • Qifeng Li +2
  • Research Article

MKGFA: Multimodal Knowledge Graph Construction and Fact-Assisted Reasoning for VQA

  • Dec 30, 2024
  • International Journal of Computational Intelligence and Applications
  • Longbao Wang +6
  • Research Article
  • Citations74

KnowIT VQA: Answering Knowledge-Based Questions about Videos

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Noa Garcia +3
  • Conference Article
  • Citations6

Sequential Visual Reasoning for Visual Question Answering

  • Nov 01, 2018
  • Jinlai Liu +3
  • Conference Article
  • Citations19

Interpretable Visual Reasoning via Induced Symbolic Space

  • Oct 01, 2021
  • Zhonghao Wang +6
  • Video Transcripts

Crake: Causal-Enhanced Table-Filler for Question Answering over Large Scale Knowledge Base

  • Jun 27, 2022
  • Underline Science Inc.
  • Yanzeng Li +3
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.