• Home
  • Search
  • Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
  • Cite Icon8
  • https://doi.org/10.3389/fncom.2022.789253Copy DOI Icon

Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks

Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. Specifically, by biologically plausible training we mean (i) all updates of weights are based on current activities of pre-synaptic units and current, or activity retrieved from short term memory of post synaptic units, including at the top-most error computing layer, (ii) complex computations such as normalization, inner products and division are avoided, (iii) asymmetric connections between units, and (iv) most learning is carried out in an unsupervised manner. SSL with a contrastive loss satisfies the third condition as it does not require labeled data and it introduces robustness to observed perturbations of objects, which occur naturally as objects or observers move in 3D and with variable lighting over time. We propose a contrastive hinge based loss whose error involves simple local computations satisfying (ii), as opposed to the standard contrastive losses employed in the literature, which do not lend themselves easily to implementation in a network architecture due to complex computations involving ratios and inner products. Furthermore, we show that learning can be performed with one of two more plausible alternatives to backpropagation that satisfy conditions (i) and (ii). The first is difference target propagation (DTP), which trains network parameters using target-based local losses and employs a Hebbian learning rule, thus overcoming the biologically implausible symmetric weight problem in backpropagation. The second is layer-wise learning, where each layer is directly connected to a layer computing the loss error. The layers are either updated sequentially in a greedy fashion (GLL) or in random order (RLL), and each training stage involves a single hidden layer network. Backpropagation through one layer needed for each such network can either be altered with fixed random feedback weights (RF) or using updated random feedback weights (URF) as in Amity's study 2019. Both methods represent alternatives to the symmetric weight issue of backpropagation. By training convolutional neural networks (CNNs) with SSL and DTP, GLL or RLL, we find that our proposed framework achieves comparable performance to standard BP learning downstream linear classifier evaluation of the learned embeddings.

Loading PDF

Similar Papers
  • Book Chapter
  • Citations179

Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik’s Cube

  • Jan 01, 2019
  • Xinrui Zhuang +5
  • Conference Article
  • Citations9

Lung Cancer Classification and Detection Using Convolutional Neural Networks

  • Sep 14, 2020
  • Deniz Nisham Anwer +1
  • Research Article
  • Citations1

Deep Learning based Effective Steganalysis

  • Mar 15, 2020
  • International Journal of Innovative Technology and Exploring Engineering
  • S Tan +85
  • Research Article
  • Citations1

SCRM-Net: Self-Supervised Deep Clustering Feature Representation for Urban 3D Mesh Semantic Segmentation

  • Apr 16, 2025
  • Remote Sensing
  • Jiahui Wang +4
  • PDF
  • Research Article
  • Citations7

DEEP CONVOLUTION NEURAL NETWORKS WITH RESNET ARCHITECTURE FOR SPECTRAL-SPATIAL CLASSIFICATION OF DRONE BORNE AND GROUND BASED HIGH RESOLUTION HYPERSPECTRAL IMAGERY

  • May 30, 2022
  • The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
  • A Galodha +3
  • Research Article
  • Citations81

Trends in the application of deep learning networks in medical image analysis: Evolution between 2012 and 2020

  • Nov 24, 2021
  • European Journal of Radiology
  • Lu Wang +9
  • PDF
  • Research Article
  • Citations4

Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks.

  • Sep 23, 2020
  • PLOS ONE
  • Nolan Peter Shaw +2
  • Research Article
  • Citations1

Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks

  • Sep 23, 2020
  • PLoS ONE
  • Nolan Peter Shaw +3
  • Research Article
  • Citations5

Classification Images Reveal that Deep Learning Networks Fail to Perceive Illusory Contours

  • Aug 31, 2017
  • Journal of Vision
  • Philip Kellman +3
  • Conference Article
  • Citations4

Helicobacter Pylori Infection Classification Based on Convolutional Neural Network and Self-Supervised Learning

  • Jun 25, 2021
  • Guo-Zhang Jian +3
  • Research Article
  • Citations130

Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images

  • Aug 04, 2017
  • Structural Control and Health Monitoring
  • Yang Xu +6
  • Research Article
  • Citations35

Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks.

  • Jun 30, 2023
  • Cognitive neurodynamics
  • Md Hasin Raihan Rabbani +1
  • Research Article
  • Citations13

Automating the Detection of Dynamically Triggered Earthquakes via a Deep Metric Learning Algorithm

  • Jan 02, 2020
  • Seismological Research Letters
  • Vivian Tang +4
  • Research Article
  • Citations7

Deep ensemble models for speech emotion classification

  • Feb 15, 2023
  • Microprocessors and Microsystems
  • Sheena Christabel Pravin +2
  • Research Article
  • Citations37

Combined deep belief network in deep learning with affinity propagation clustering algorithm for roller bearings fault diagnosis without data label

  • Jul 04, 2018
  • Journal of Vibration and Control
  • Fan Xu +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.