• Home
  • Search
  • Self-supervised visual learning in the low-data regime: A comparative evaluation
  • Cite Icon20
  • https://doi.org/10.1016/j.neucom.2024.129199Copy DOI Icon

Self-supervised visual learning in the low-data regime: A comparative evaluation

Show More
  • Abstract
  • Highlights & Summary
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining on a ‘pretext task’ that does not require ground-truth labels/annotation. This allows efficient representation learning from massive amounts of unlabeled training data, which in turn leads to increased accuracy in a ‘downstream task’ by exploiting supervised transfer learning. Despite the relatively straightforward conceptualization and applicability of SSL, it is not always feasible to collect and/or to utilize very large pretraining datasets, especially when it comes to real-world application settings. In particular, in cases of specialized and domain-specific application scenarios, it may not be achievable or practical to assemble a relevant image pretraining dataset in the order of millions of instances or it could be computationally infeasible to pretrain at this scale, e.g., due to unavailability of sufficient computational resources that SSL methods typically require to produce improved visual analysis results. This situation motivates an investigation on the effectiveness of common SSL pretext tasks, when the pretraining dataset is of relatively limited/constrained size. This work briefly introduces the main families of modern visual SSL methods and, subsequently, conducts a thorough comparative experimental evaluation in the low-data regime, targeting to identify: (a) what is learnt via low-data SSL pretraining, and (b) how do different SSL categories behave in such training scenarios. Interestingly, for domain-specific downstream tasks, in-domain low-data SSL pretraining outperforms the common approach of large-scale pretraining on general datasets. Grounded on the obtained results, valuable insights are highlighted regarding the performance of each category of SSL methods, which in turn suggest straightforward future research directions in the field.

Similar Papers
  • Research Article
  • Citations14

Benchmarking Self-Supervised Contrastive Learning Methods for Image-Based Plant Phenotyping

  • Jan 01, 2023
  • Plant Phenomics
  • Franklin C Ogidi +2
  • Conference Article
  • Citations2

A Novel Multi-Task Self-Supervised Representation Learning Paradigm

  • May 28, 2021
  • Control theory & applications
  • Yinggang Li +5
  • PDF
  • Research Article
  • Citations16

A Generic Self-Supervised Learning (SSL) Framework for Representation Learning from Spectral–Spatial Features of Unlabeled Remote Sensing Imagery

  • Nov 03, 2023
  • Remote Sensing
  • Xin Zhang +1
  • Book Chapter
  • Citations7

CaSS: A Channel-Aware Self-supervised Representation Learning Framework for Multivariate Time Series Classification

  • Jan 01, 2022
  • Yijiang Chen +4
  • Research Article
  • Citations8

Self-supervised learning for heterogeneous graph via structure information based on metapath

  • May 12, 2023
  • Applied Soft Computing
  • Shuai Ma +2
  • Conference Article
  • Citations4

Conditional Independence for Pretext Task Selection in Self-Supervised Speech Representation Learning

  • Aug 30, 2021
  • Salah Zaiem +2
  • Conference Article
  • Citations4

Self-supervised Learning Based on Max-tree Representation for Medical Image Segmentation

  • Jul 18, 2022
  • Qian Tang +2
  • Research Article
  • Citations2063

Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey.

  • May 04, 2020
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Longlong Jing +1
  • Conference Article
  • Citations18

Self-supervised Visual Attribute Learning for Fashion Compatibility

  • Oct 01, 2021
  • Donghyun Kim +5
  • Conference Article

Self-supervised multi-task learning for semantic segmentation of urban scenes

  • Sep 12, 2021
  • Jonathan Gonzalez-Santiago +2
  • PDF
  • Research Article
  • Citations2

PatchLoc: Embedded Patch Localization Pretext Task for Tumor Segmentation in Medical Images

  • Jan 01, 2024
  • IEEE Access
  • Ramchandra Cheke +3
  • Research Article
  • Citations2

Research on Self-Supervised Comparative Learning for Computer Vision

  • Aug 17, 2021
  • Journal of Electronic Research and Application
  • Yuanyuan Liu +1
  • Research Article
  • Citations48

CS-CO: A Hybrid Self-Supervised Visual Representation Learning Method for H&E-stained Histopathological Images.

  • Oct 01, 2022
  • Medical Image Analysis
  • Pengshuai Yang +6
  • Research Article
  • Citations16

Pretext Tasks Selection for Multitask Self-Supervised Audio Representation Learning

  • Oct 01, 2022
  • IEEE Journal of Selected Topics in Signal Processing
  • Salah Zaiem +3
  • Conference Article
  • Citations64

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting

  • Jun 01, 2021
  • Ayan Kumar Bhunia +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.