• Home
  • Search
  • VIG: Visual Information-Guided Knowledge-Based Visual Question Answering
  • https://doi.org/10.1109/cscwd61410.2024.10580527Copy DOI Icon

VIG: Visual Information-Guided Knowledge-Based Visual Question Answering

  • May 8, 2024
  • Heng Liu +5 more
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

A better knowledge-based visual question answering (KBVQA) model needs to rely on visual features, question features, and related external knowledge to solve an open visual question answering task. Although the existing knowledge-based visual question answering works have achieved some accomplishments, there are still the following challenges: 1) There is a serious lack of visual feature information. Image information is worth a thousand words. Only relying on the converted salient text information is difficult to express the original rich information of the image. 2) The external knowledge acquired is not comprehensive enough, and there is a lack of relevant knowledge directly retrieved by visual feature information. To solve these challenges, we propose a Visual Information-Guided knowledge-based visual question answering (VIG) model. It fully considers the utilization of visual features information. Specifically: 1) We introduce multi-granularity visual information that can comprehensively characterize visual feature information. 2) We consider not only the knowledge retrieved through text information but also the knowledge directly retrieved from visual feature information. Finally, we feed the visual features and retrieved multiple text knowledge into an encoder-decoder module to generate an answer. We perform extensive experiments on the OKVQA dataset and achieve state-of-the-art performance of 60.27% accuracy.

Similar Papers
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Conference Article
  • Citations7

Explicit Knowledge Integration for Knowledge-Aware Visual Question Answering about Named Entities

  • Jun 12, 2023
  • Omar Adjali +4
  • Conference Article
  • Citations11

Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering

  • Jan 01, 2024
  • Junnan Dong +5
  • PDF
  • Conference Article
  • Citations27

Improving Visual Question Answering by Referring to Generated Paragraph Captions

  • Jan 01, 2019
  • Hyounghun Kim +1
  • PDF
  • Research Article
  • Citations1

Learning to Reason on Tree Structures for Knowledge-Based Visual Question Answering

  • Feb 17, 2022
  • Sensors (Basel, Switzerland)
  • Qifeng Li +2
  • Conference Article
  • Citations677

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

  • Jun 01, 2019
  • Kenneth Marino +3
  • Conference Article
  • Citations2

Estimating Viewed Images with Natural Language Question Answering from fMRI Data

  • Mar 01, 2020
  • Saya Takada +3
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • Research Article
  • Citations9

Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering

  • Oct 23, 2023
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Yan Wang +6
  • Research Article
  • Citations116

Cross-modal knowledge reasoning for knowledge-based visual question answering

  • Jul 22, 2020
  • Pattern Recognition
  • Jing Yu +5
  • Research Article
  • Citations64

Image captioning for effective use of language models in knowledge-based visual question answering

  • Aug 28, 2022
  • Expert Systems with Applications
  • Ander Salaberria +4
  • Research Article
  • Citations265

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

  • Jun 28, 2022
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Zhengyuan Yang +6
  • PDF
  • Conference Article
  • Citations56

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

  • Jan 01, 2021
  • Man Luo +3
  • Book Chapter
  • Citations96

Question-Guided Hybrid Convolution for Visual Question Answering

  • Jan 01, 2018
  • Peng Gao +6
  • PDF
  • Research Article
  • Citations17

Knowledge-Based Visual Question Answering Using Multi-Modal Semantic Graph

  • Mar 14, 2023
  • Electronics
  • Lei Jiang +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.