• Home
  • Search
  • Confidence-aware Non-repetitive Multimodal Transformers for TextCaps
  • Cite Icon19
  • https://doi.org/10.1609/aaai.v35i4.16389Copy DOI Icon

Confidence-aware Non-repetitive Multimodal Transformers for TextCaps

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

When describing an image, reading text in the visual scene is crucial to understand the key information. Recent work explores the TextCaps task, i.e. image captioning with reading Optical Character Recognition (OCR) tokens, which requires models to read text and cover them in generated captions. Existing approaches fail to generate accurate descriptions because of their (1) poor reading ability; (2) inability to choose the crucial words among all extracted OCR tokens; (3) repetition of words in predicted captions. To this end, we propose a Confidence-aware Non-repetitive Multimodal Transformers (CNMT) to tackle the above challenges. Our CNMT consists of a reading, a reasoning and a generation modules, in which Reading Module employs better OCR systems to enhance text reading ability and a confidence embedding to select the most noteworthy tokens. To address the issue of word redundancy in captions, our Generation Module includes a repetition mask to avoid predicting repeated word in captions. Our model outperforms state-of-the-art models on TextCaps dataset, improving from 81.0 to 93.0 in CIDEr. Our source code is publicly available.

Similar Papers
  • Research Article
  • Citations1

Childhood Reading Ability and Pain in Childhood Through to Midlife

  • Apr 01, 2024
  • The Journal of Pain
  • Elinor C Bridges +3
  • Research Article
  • Citations29

Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa

  • Jan 21, 2023
  • Pattern Recognition
  • Gangyan Zeng +7
  • PDF
  • Research Article
  • Citations3

Clinical profile of reading ability and reading and writing achievement of children with borderline full-scale intellectual quotient: a prospective study

  • Sep 07, 2021
  • BMC Pediatrics
  • Riyo Ueda +5
  • Research Article
  • Citations1

Depression, Generalize Anxiety Disorder, and School Avoidance in Early School Years in Pupils with Poor Reading Ability

  • Aug 15, 2023
  • Journal of the Medical Association of Thailand
  • Research Article
  • Citations58

Structured Multimodal Attentions for TextVQA.

  • Dec 01, 2022
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Chenyu Gao +6
  • Research Article
  • Citations25

Cursive-Text: A Comprehensive Dataset for End-to-End Urdu Text Recognition in Natural Scene Images

  • May 21, 2020
  • Data in Brief
  • Asghar Ali Chandio +3
  • Research Article
  • Citations5

A Deep Learning-Based Text Detection and Recognition Approach for Natural Scenes

  • Oct 27, 2022
  • Journal of Circuits, Systems and Computers
  • Xuexiang Li
  • Conference Article
  • Citations138

TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

  • Jun 01, 2021
  • Amanpreet Singh +5
  • Research Article
  • Citations3

Performance Evaluation of Efficient and Accurate Text Detection and Recognition in Natural Scenes Images Using EAST and OCR Fusion

  • Jan 01, 2025
  • International Journal of Advanced Computer Science and Applications
  • Vishnu Kant Soni +5
  • Research Article
  • Citations45

Executive Dysfunction in Poor Readers Born Prematurely at High Risk

  • May 11, 2009
  • Developmental Neuropsychology
  • Richard E Frye +3
  • Conference Article
  • Citations5

Scene text detection and recognition system for visually impaired people in real world

  • Oct 09, 2018
  • Lei Fei +5
  • Supplementary Content
  • Citations3

Text-detection and -recognition from natural images

  • Feb 10, 2020
  • Figshare
  • Hanaa F Mahmood
  • Conference Article
  • Citations4

Research on Text Recognition of Natural Scenes for Complex Situations

  • Jul 22, 2022
  • Wenhua Yu +2
  • Conference Article
  • Citations6

Vietnamese Scene Text Detection and Recognition using Deep Learning: An Empirical Study

  • Jul 29, 2022
  • Nhat Truong Pham +5
  • Research Article
  • Citations117

Scene text detection and recognition with advances in deep learning: a survey

  • Mar 27, 2019
  • International Journal on Document Analysis and Recognition (IJDAR)
  • Xiyan Liu +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.