• Open Access IconOpen Access
  • Cite Icon38
  • https://doi.org/10.1145/3474085.3475606Copy DOI Icon

Beyond OCR + VQA

  • Oct 17, 2021
  • Gangyan Zeng +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Text-based visual question answering (TextVQA) requires analyzing both the visual contents and texts in an image to answer a question, which is more practical than general visual question answering (VQA). Existing efforts tend to regard optical character recognition (OCR) as a pre-processing and then combine it with a VQA framework. It makes the performance of multimodal reasoning and question answering highly depend on the accuracy of OCR. In this work, we address this issue with two perspectives. First, we take advantages of multimodal cues to complete the semantic information of texts. A visually enhanced text embedding is proposed to enable understanding of texts without accurately recognizing them. Second, we further leverage rich contextual information to modify the answer texts even if the OCR module does not correctly recognize them. In addition, the visual objects are endued with semantic representations to enable objects in the same semantic space as OCR tokens. Equipped with these techniques, the cumulative error propagation caused by poor OCR performance is effectively suppressed. Extensive experiments on TextVQA and ST-VQA datasets demonstrate that our approach achieves the state-of-the-art performance in terms of accuracy and robustness.

Similar Papers
  • Research Article
  • Citations29

Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa

  • Jan 21, 2023
  • Pattern Recognition
  • Gangyan Zeng +7
  • Book Chapter
  • Citations18

Building Super-Resolution Image Generator for OCR Accuracy Improvement

  • Jan 01, 2020
  • Xujun Peng +1
  • Conference Article
  • Citations59

Cascade Reasoning Network for Text-based Visual Question Answering

  • Oct 12, 2020
  • Fen Liu +5
  • Conference Article
  • Citations1

Text-based visual question answering with knowledge base

  • Mar 07, 2021
  • Fang Zhou +4
  • Research Article
  • Citations3

Improving Tesseract OCR Accuracy Using SymSpell Algorithm on Passport Data

  • Jan 27, 2025
  • Sinkron
  • Iqbaluddin Syam Had +2
  • Research Article
  • Citations300

Optical Character Recognition by Open source OCR Tool Tesseract: A Case Study

  • Oct 20, 2012
  • International Journal of Computer Applications
  • Chirag Patel +2
  • Research Article
  • Citations6

Reading in the mist: high-quality optical character recognition based on freely available early modern digitized books

  • Apr 06, 2022
  • Digital Scholarship in the Humanities
  • Andrea Sangiacomo +4
  • PDF
  • Research Article
  • Citations2

Pixel-Level Degradation for Text Image Super-Resolution and Recognition

  • Nov 05, 2023
  • Electronics
  • Xiaohong Qian +4
  • Conference Article
  • Citations4

Feature Extraction Method of Machine Translation Equivalent Pairs in Chinese-English Comparable Corpus based OCR Recognition

  • Jun 03, 2021
  • Bo Wang
  • Research Article
  • Citations6

Deep Learning-Based Multifunctional End-to-End Model for Optical Character Classification and Denoising

  • Nov 15, 2023
  • Journal of Computational Methods in Engineering Applications
  • Shuguang Xiong +2
  • Conference Article
  • Citations1

Issue Based OCR Error Prediction in Video Streams

  • Sep 23, 2020
  • Dirk Siegmund +2
  • Conference Article
  • Citations63

A Fast Alignment Scheme for Automatic OCR Evaluation of Books

  • Jan 01, 2011
  • Ismet Zeki Yalniz +1
  • Conference Article
  • Citations1

Towards a Higher Accuracy of Optical Character Recognition of Chinese Rare Books in Making Use of Text Model

  • May 08, 2019
  • Hsiang-An Wang +1
  • Research Article
  • Citations2

A conclusive methodology for rating OCR performance

  • Jul 26, 2005
  • Journal of the American Society for Information Science and Technology
  • Nathan E Brener +2
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.