• Home
  • Search
  • Robust Transfer Learning with Pretrained Language Models through Adapters
  • https://doi.org/10.48448/1bqq-8b16Copy DOI Icon

Robust Transfer Learning with Pretrained Language Models through Adapters

Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

Similar Papers
  • PDF
  • Research Article
  • Citations10

AGI-P: A Gender Identification Framework for Authorship Analysis Using Customized Fine-Tuning of Multilingual Language Model

  • Jan 01, 2024
  • IEEE Access
  • Raheem Sarwar +6
  • Conference Article
  • Citations3

Chinese-Korean Weibo Sentiment Classification Based on Pre-trained Language Model and Transfer Learning

  • May 06, 2022
  • Hengxuan Wang +3
  • Research Article
  • Citations1

Using a large language model to provide individualized feedback for pre-service physics teachers’ written reflections

  • Nov 21, 2025
  • Disciplinary and Interdisciplinary Science Education Research
  • Stefan Sorge +2
  • Research Article
  • Citations14

Fine-Tuned Understanding: Enhancing Social Bot Detection With Transformer-Based Classification

  • Jan 01, 2024
  • IEEE Access
  • Amine Sallah +6
  • PDF
  • Conference Article
  • Citations31

PANLP at MEDIQA 2019: Pre-trained Language Models, Transfer Learning and Knowledge Distillation

  • Jan 01, 2019
  • Wei Zhu +6
  • PDF
  • Research Article
  • Citations1

Prompt Learning with Structured Semantic Knowledge Makes Pre-Trained Language Models Better

  • Jul 30, 2023
  • Electronics
  • Hai-Tao Zheng +5
  • Research Article
  • Citations2

Adaptive Boosting LLMs for Text Classification.

  • Jan 01, 2026
  • IEEE transactions on neural networks and learning systems
  • Mengyao Wang +6
  • Research Article
  • Citations13

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration

  • May 16, 2022
  • Knowledge-Based Systems
  • Chen Ye +6
  • Research Article
  • Citations16

A Review of the Opportunities and Challenges with Large Language Models in Radiology: The Road Ahead.

  • Nov 21, 2024
  • AJNR. American journal of neuroradiology
  • Neetu Soni +4
  • Conference Article

Exploring Layer-wise Representations of English and Chinese Homonymy in Pre-trained Language Models

  • Jan 01, 2025
  • Matthew King-Hang Ma +3
  • PDF
  • Research Article
  • Citations10

Classification and analysis of text transcription from Thai depression assessment tasks among patients with depression.

  • Mar 30, 2023
  • PLOS ONE
  • Adirek Munthuli +8
  • Supplementary Content

Deep learning techniques for geospatial data integration, knowledge mining, and real-world applications

  • Jan 01, 2025
  • Pasquale Balsebre
  • Conference Article
  • Citations9

On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code

  • Nov 30, 2023
  • Martin Weyssow +4
  • Conference Article
  • Citations16

Unsupervised Neural Machine Translation for English to Kannada Using Pre-Trained Language Model

  • Oct 03, 2022
  • Shailashree K Sheshadri +4
  • Research Article
  • Citations1

An Empirical Study on Language Models for Generating Log Statements in Test Code

  • Aug 12, 2025
  • ACM Transactions on Software Engineering and Methodology
  • Honglin Shu +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.