• Home
  • Search
  • Self-Supervised Deep Learning Models for Low-Resource NLP Applications
  • https://doi.org/10.71143/5nh5zh32Copy DOI Icon

Self-Supervised Deep Learning Models for Low-Resource NLP Applications

  • Abstract
  • Literature Map
  • Similar Papers
Abstract

The availability of huge annotated datasets and deep learning have led to astounding Natural Language Processing (NLP) progress. However, there is a plethora of low-resource languages, lack the labelled corpora to train the models in the supervision. This kind of digital divide does not allow fair NLP development across language groups. Self-supervised learning (SSL) is a new paradigm in recent years that leverages large amounts of untagged text to learn powerful representations with little or no manual annotation. This paper is a review of self-supervised deep learning models in low-resource NLP tasks. It begins with the definition of the principles of the SSL and the difference between this approach and supervised and unsupervised approaches. We describe why methods such as masked language modeling, contrastive learning, and autoregressive modeling underlie modern pre-trained transformers such as BERT, GPT, and mBERT. A particular interest is paid to multilingual and cross lingual SSL schemes which allow knowledge transfer between high-resource and low resource languages. Low-resource tasks, including machine translation, sentiment analysis, speech-to-text, and information retrieval are reviewed. As per benchmark studies, even in small labelled samples, the SSL models can achieve big gains in accuracy and generalization. Computation cost, representational bias and morphologically rich and under-documented language evaluation, are however, problematic. The researchers observe that the study suffers certain limitations, such as over-reliance on high-resource pretraining information, inequity between linguistic groups, and the difficulty of deploying large-scale SSL models in resource-constrained circumstances. The future directions include lightweight multilingual models, federation of learning in NLP and symbolic linguistic knowledge and combination with SSL. Self-supervised deep learning bridges an essential gap between the high-resource and low-resource languages, and would be a highly valuable step toward inclusive, global NLP innovation.

Similar Papers
  • Research Article

Abstract 5059: Self-supervised representation learning of somatic mutational data

  • Apr 21, 2025
  • Cancer Research
  • Marzieh Haghighi +2
  • Research Article
  • Citations28

Improving quantitative MRI using self-supervised deep learning with model reinforcement: Demonstration for rapid T1 mapping.

  • Feb 11, 2024
  • Magnetic resonance in medicine
  • Wanyu Bian +2
  • Research Article
  • Citations2

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

  • Jun 13, 2024
  • Knowledge-Based Systems
  • Zhensheng Shi +2
  • PDF
  • Research Article
  • Citations17

ProteinMAE: masked autoencoder for protein surface self-supervised learning.

  • Nov 29, 2023
  • Bioinformatics
  • Mingzhi Yuan +6
  • Research Article

P277 Predicting disease relevant features in Crohn’s Disease and Ulcerative Colitis from Haematoxylin & Eosin stained whole slide images using self-supervised deep learning

  • Jan 30, 2023
  • Journal of Crohn's and Colitis
  • R Mokhtari +11
  • Conference Article
  • Citations8

A General Framework for Inverse Problem Solving using Self-Supervised Deep Learning: Validations in Ultrasound and Photoacoustic Image Reconstruction

  • Sep 11, 2021
  • Jingke Zhang +4
  • Research Article
  • Citations5

Self-supervised learning for MRI reconstruction: a review and new perspective.

  • Jun 26, 2025
  • Magma (New York, N.Y.)
  • Xinzhen Li +3
  • Research Article
  • Citations350

A comparison review of transfer learning and self-supervised learning: Definitions, applications, advantages and limitations

  • Dec 02, 2023
  • Expert Systems with Applications
  • Zehui Zhao +4
  • Research Article
  • Citations26

Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs

  • Oct 08, 2024
  • Proceedings of the ACM on Programming Languages
  • Federico Cassano +9
  • Research Article
  • Citations7

Advanced montane bird monitoring using self-supervised learning and transformer on passive acoustic data

  • Dec 01, 2024
  • Ecological Informatics
  • Yu-Cheng Wei +4
  • Conference Article

Exploring the Depths: A Comprehensive Survey of Neural Networks and their Applications

  • Mar 17, 2025
  • AIJR Proceedings
  • Harshit Gupta +4
  • Research Article
  • Citations4

Foundation models for electrocardiogram interpretation: clinical implications

  • Jan 22, 2026
  • European Heart Journal
  • Alexis Nolin-Lapalme +32
  • Research Article
  • Citations1

Representation-based ECG signal prediction forneural networks pre-training

  • Apr 04, 2025
  • Herald of Advanced Information Technology
  • Serhii G Stavychenko +1
  • Conference Article
  • Citations4

GAUSS: GrAph-customized Universal Self-Supervised Learning

  • May 13, 2024
  • Liang Yang +9
  • Conference Article
  • Citations8

A Self-supervised Deep Learning Network for Low-Dose CT Reconstruction

  • Nov 01, 2018
  • Kaichao Liang +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.