• Home
  • Search
  • Improving Visual Question Answering by Referring to Generated Paragraph Captions
  • Cite Icon27
  • https://doi.org/10.18653/v1/p19-1351Copy DOI Icon

Improving Visual Question Answering by Referring to Generated Paragraph Captions

  • Jan 1, 2019
  • Hyounghun Kim +1 more
Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Paragraph-style image captions describe diverse aspects of an image as opposed to the more common single-sentence captions that only provide an abstract description of the image. These paragraph captions can hence contain substantial information of the image for tasks such as visual question answering. Moreover, this textual information is complementary with visual information present in the image because it can discuss both more abstract concepts and more explicit, intermediate symbolic information about objects, events, and scenes that can directly be matched with the textual question and copied into the textual answer (i.e., via easier modality match). Hence, we propose a combined Visual and Textual Question Answering (VTQA) model which takes as input a paragraph caption as well as the corresponding image, and answers the given question based on both inputs. In our model, the inputs are fused to extract related information by cross-attention (early fusion), then fused again in the form of consensus (late fusion), and finally expected answers are given an extra score to enhance the chance of selection (later fusion). Empirical results show that paragraph captions, even when automatically generated (via an RL-based encoder-decoder model), help correctly answer more visual questions. Overall, our joint model, when trained on the Visual Genome dataset, significantly improves the VQA performance over a strong baseline model.

Loading PDF

Similar Papers
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Conference Article
  • Citations2

Estimating Viewed Images with Natural Language Question Answering from fMRI Data

  • Mar 01, 2020
  • Saya Takada +3
  • Research Article
  • Citations17

Inner Knowledge-based Img2Doc Scheme for Visual Question Answering

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Qun Li +4
  • PDF
  • Research Article
  • Citations2

Visual Question Answering for Intelligent Interaction

  • Jul 06, 2022
  • Mobile Information Systems
  • Panpan Gao +4
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
  • Conference Article
  • Citations211

Visual Question Generation as Dual Task of Visual Question Answering

  • Jun 01, 2018
  • Yikang Li +6
  • Conference Article
  • Citations29

Contrast and Classify: Training Robust VQA Models

  • Oct 01, 2021
  • Yash Kant +4
  • Conference Article

VIG: Visual Information-Guided Knowledge-Based Visual Question Answering

  • May 08, 2024
  • Heng Liu +5
  • Conference Article
  • Citations90

Multi-Modality Latent Interaction Network for Visual Question Answering

  • Oct 01, 2019
  • Gao Peng +4
  • Conference Article
  • Citations603

VizWiz Grand Challenge: Answering Visual Questions from Blind People

  • Jun 01, 2018
  • Danna Gurari +7
  • Book Chapter
  • Citations1

Towards Open Ended and Free Form Visual Question Answering: Modeling VQA as a Factoid Question Answering Problem

  • Jan 01, 2021
  • Abhishek Narayanan +3
  • Conference Article

Square peg, round hole: A case study on using Visual Question & Answering in Games

  • Oct 15, 2021
  • Paulo Bala +4
  • Book Chapter
  • Citations1

Automated Question Generation and Answer Verification Using Visual Data

  • Jan 01, 2020
  • Shrey Nahar +4
  • Conference Article
  • Citations59

Visual Question Answering as Reading Comprehension

  • Jun 01, 2019
  • Hui Li +3
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.