• Home
  • Search
  • Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa
  • Cite Icon29
  • https://doi.org/10.1016/j.patcog.2023.109337Copy DOI Icon

Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Text-based visual question answering (TextVQA), which answers a visual question by considering both visual contents and scene texts, has attracted increasing attention recently. Most existing methods employ an optical character recognition (OCR) module as a pre-processor to read texts, then combine it with a visual question answering (VQA) framework. However, inaccurate OCR results may lead to cumulative error propagation, and the correlation between text reading and text-based reasoning is not fully exploited. In this work, we integrate OCR into the flow of TextVQA, targeting the mutual reinforcement of OCR and VQA tasks. Specifically, a visually enhanced text embedding module is proposed to predict semantic features from the visual information of texts, by which texts can be reasonably understood even without accurate recognition. Further, two elaborate schemes are developed to leverage contextual information in VQA to modify OCR results. The first scheme is a reading modification module that adaptively selects the answer results according to the contexts. Second, we propose an efficient end-to-end text reading and reasoning network, where the downstream VQA signal contributes to the optimization of text reading. Extensive experiments show that our method outperforms existing alternatives in terms of accuracy and robustness, whether ground truth OCR annotations are used or not.

Similar Papers
  • Conference Article
  • Citations227

OCR-VQA: Visual Question Answering by Reading Text in Images

  • Sep 01, 2019
  • Anand Mishra +3
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
  • Conference Article
  • Citations1

Multi-modal Domain Adaptation for Text Visual Question Answering Tasks

  • Nov 28, 2023
  • Zhiyuan Li +2
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Conference Article
  • Citations211

Visual Question Generation as Dual Task of Visual Question Answering

  • Jun 01, 2018
  • Yikang Li +6
  • Conference Article
  • Citations2

Estimating Viewed Images with Natural Language Question Answering from fMRI Data

  • Mar 01, 2020
  • Saya Takada +3
  • Conference Article
  • Citations48

Coarse-to-Fine Reasoning for Visual Question Answering

  • Jun 01, 2022
  • Binh X Nguyen +5
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Conference Article
  • Citations38

Beyond OCR + VQA

  • Oct 17, 2021
  • Gangyan Zeng +3
  • Conference Article
  • Citations4

Estimation Of Visual Contents Based On Question Answering From Human Brain Activity

  • Oct 01, 2020
  • Saya Takada +3
  • Research Article
  • Citations58

Structured Multimodal Attentions for TextVQA.

  • Dec 01, 2022
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Chenyu Gao +6
  • Research Article
  • Citations19

Achieving Human Parity on Visual Question Answering

  • Apr 04, 2023
  • ACM Transactions on Information Systems
  • Ming Yan +11
  • Conference Article
  • Citations31

Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling

  • Oct 01, 2021
  • Xiaopeng Lu +4
  • Book Chapter
  • Citations3

Optimal Image Feature Ranking and Fusion for Visual Question Answering

  • Sep 09, 2020
  • Sruthy Manmadhan +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.