• Home
  • Search
  • Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models
  • Cite Icon2
  • https://doi.org/10.1609/aaai.v37i2.25357Copy DOI Icon

Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Alignment between image and text has shown promising improvements on patch-level pre-trained document image models. However, investigating more effective or finer-grained alignment techniques during pre-training requires a large amount of computation cost and time. Thus, a question naturally arises: Could we fine-tune the pre-trained models adaptive to downstream tasks with alignment objectives and achieve comparable or better performance? In this paper, we propose a new model architecture with alignment-enriched tuning (dubbed AETNet) upon pre-trained document image models, to adapt downstream tasks with the joint task-specific supervised and alignment-aware contrastive objective. Specifically, we introduce an extra visual transformer as the alignment-ware image encoder and an extra text transformer as the alignment-ware text encoder before multimodal fusion. We consider alignment in the following three aspects: 1) document-level alignment by leveraging the cross-modal and intra-modal contrastive loss; 2) global-local alignment for modeling localized and structural information in document images; and 3) local-level alignment for more accurate patch-level information. Experiments on various downstream tasks show that AETNet can achieve state-of-the-art performance on various downstream tasks. Notably, AETNet consistently outperforms state-of-the-art pre-trained models, such as LayoutLMv3 with fine-tuning techniques, on three different downstream tasks. Code is available at https://github.com/MAEHCM/AET.

Similar Papers
  • Conference Article

GTSP: A Unified Framework for Graph Pretraining and Task-Specific Prompting

  • Sep 19, 2025
  • Fanghua Lu
  • Conference Article
  • Citations9

On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code

  • Nov 30, 2023
  • Martin Weyssow +4
  • Research Article
  • Citations4

WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter

  • Jul 09, 2024
  • Knowledge-Based Systems
  • Bin Shi +3
  • PDF
  • Research Article
  • Citations1

Prompt Learning with Structured Semantic Knowledge Makes Pre-Trained Language Models Better

  • Jul 30, 2023
  • Electronics
  • Hai-Tao Zheng +5
  • Research Article
  • Citations1

Efficient Transfer From Image-Based Large Multimodal Models to Video Tasks

  • Jan 01, 2025
  • IEEE Transactions on Multimedia
  • Shidong Cao +6
  • Research Article

Parameter-Efficient Fine-Tuning via Meta-Regularizer

  • Mar 06, 2026
  • International Journal of Computer Vision
  • Jinyoung Park +4
  • PDF
  • Research Article
  • Citations1

Removing Backdoors in Pre-trained Models by Regularized Continual Pre-training

  • Dec 20, 2023
  • Transactions of the Association for Computational Linguistics
  • Biru Zhu +9
  • PDF
  • Research Article
  • Citations4

Can using a pre-trained deep learning model as the feature extractor in the bag-of-deep-visual-words model always improve image classification accuracy?

  • Feb 29, 2024
  • PLOS ONE
  • Ye Xu +3
  • Research Article
  • Citations10

Investigation of the BERT model on nucleotide sequences with non-standard pre-training and evaluation of different k-mer embeddings.

  • Oct 03, 2023
  • Bioinformatics
  • Yao-Zhong Zhang +2
  • Research Article
  • Citations16

CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models

  • Jan 01, 2021
  • IEEE/ACM Transactions on Audio, Speech, and Language Processing
  • Yusheng Su +7
  • Book Chapter
  • Citations7

PoetryBERT: Pre-training with Sememe Knowledge for Classical Chinese Poetry

  • Jan 01, 2022
  • Jiaqi Zhao +3
  • Conference Article
  • Citations16

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

  • May 13, 2024
  • Yun Zhu +5
  • PDF
  • Research Article
  • Citations53

Enhancing coffee bean classification: a comparative analysis of pre-trained deep learning models

  • Apr 01, 2024
  • Neural Computing and Applications
  • Esraa Hassan
  • Research Article
  • Citations29

Unsupervised Adversarially Robust Representation Learning on Graphs

  • Jun 28, 2022
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Jiarong Xu +6
  • Conference Article
  • Citations1

Research on Pre-training of Tibetan Natural Language Processing

  • Jul 16, 2021
  • Zhensong Li +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.