• https://doi.org/10.1109/tpami.2021.3132034/mm1Copy DOI Icon

Supp1-3132034.pdf

  • Dec 17, 2021
  • Peng Wang
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Text based Visual Question Answering (TextVQA) is a recently raised challenge requiring models to read text in images and answer natural language questions by jointly reasoning over the question, textual information and visual content. Introduction of this new modality - Optical Character Recognition (OCR) tokens ushers in demanding reasoning requirements. Most of the state-of-the-art (SoTA) VQA methods fail when answer these questions because of three reasons: (1) poor text reading ability; (2) lack of textual-visual reasoning capacity; and (3) choosing discriminative answering mechanism over generative couterpart (although this has been further addressed by M4C). In this paper, we propose an end-to-end structured multimodal attention (SMA) neural network to mainly solve the first two issues above. SMA first uses a structural graph representation to encode the object-object, object-text and text-text relationships appearing in the image, and then designs a multimodal graph attention network to reason over it. Finally, the outputs from the above modules are processed by a global-local attentional answering module to produce an answer splicing together tokens from both OCR and general vocabulary iteratively by following M4C.

Similar Papers
  • Research Article
  • Citations58

Structured Multimodal Attentions for TextVQA.

  • Dec 01, 2022
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Chenyu Gao +6
  • Research Article
  • Citations29

Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa

  • Jan 21, 2023
  • Pattern Recognition
  • Gangyan Zeng +7
  • Research Article
  • Citations1

Childhood Reading Ability and Pain in Childhood Through to Midlife

  • Apr 01, 2024
  • The Journal of Pain
  • Elinor C Bridges +3
  • PDF
  • Research Article
  • Citations3

A deep action-oriented video image classification system for text detection and recognition

  • Oct 09, 2021
  • SN Applied Sciences
  • Abhra Chaudhuri +6
  • Conference Article
  • Citations38

Beyond OCR + VQA

  • Oct 17, 2021
  • Gangyan Zeng +3
  • Research Article
  • Citations9

Text detection and recognition based on a lensless imaging system.

  • May 09, 2022
  • Applied Optics
  • Yinger Zhang +6
  • Conference Article
  • Citations3

A Text Detection and Recognition System Based on an End-to-End Trainable Framework from UAV Imagery

  • Dec 01, 2018
  • Qingtian Wu +2
  • Book Chapter
  • Citations1

Video Indexing System Based on Multimodal Information Extraction Using Combination of ASR and OCR

  • Jan 01, 2022
  • Sandeep Varma +4
  • PDF
  • Research Article
  • Citations3

Text Detection and Recognition for X-ray Weld Seam Images

  • Mar 13, 2024
  • Applied Sciences
  • Qihang Zheng +1
  • Research Article
  • Citations17

Text detection and recognition in images and video sequences

  • Jan 01, 2003
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne)
  • Datong Chen
  • Research Article
  • Citations19

Confidence-aware Non-repetitive Multimodal Transformers for TextCaps

  • May 18, 2021
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Zhaokai Wang +3
  • Research Article

Hybrid model for Chinese character recognition based on Tesseract-OCR

  • Jan 01, 2020
  • International Journal of Internet Protocol Technology
  • Yi Wei Ma +2
  • Conference Article
  • Citations14

Applications of Text Detection and its Challenges

  • Aug 10, 2015
  • M P Nevetha +1
  • Research Article
  • Citations18

Robust visual question answering via polarity enhancement and contrast

  • Nov 01, 2024
  • Neural Networks
  • Dahe Peng +1
  • Book Chapter
  • Citations1

TORA: Text Summarization Using Optical Character Recognition and Attention Neural Networks

  • Nov 16, 2021
  • H R Sneha +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.