• Home
  • Search
  • SVRG-AALR: Stochastic Variance-Reduced Gradient Method with Adaptive Alternating Learning Rate for Training Deep Neural Networks
  • https://doi.org/10.3390/electronics14152979Copy DOI Icon

SVRG-AALR: Stochastic Variance-Reduced Gradient Method with Adaptive Alternating Learning Rate for Training Deep Neural Networks

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

The stochastic variance-reduced gradient (SVRG) theory is particularly well-suited for addressing gradient variance in deep neural network (DNN) training; however, its direct application to DNN training is hindered by adaptation challenges. To tackle this issue, the present paper proposes a series of strategies focused on adaptive alternating learning rates to effectively adapt SVRG for DNN training. Firstly, within the outer loop of SVRG, both the full gradient and the learning rate specific to DNN training are computed. For two distinct formulas used for calculating the learning rate, an alternating strategy is introduced that employs them alternately across iterations. This approach allows for simultaneous provision of diverse guidance information regarding parameter change rates and gradient change rates during DNN weight updates. Additionally, a threshold method is utilized to correct the learning rate into an appropriate range, thereby accelerating convergence. Secondly, in the inner loop of SVRG, DNN weights are updated using mini-batch average gradient along with the proposed learning rate. Concurrently, mini-batch average gradients from each iteration within the inner loop are refined and aggregated into a single gradient exhibiting reduced variance through an inertia strategy. This refined gradient is then relayed back to the outer loop to recalculate the new learning rate. The efficacy of the proposed algorithm has been validated on models including LeNet, VGG11, ResNet34, and DenseNet121 while being compared against several classic and advanced optimizers. Experimental results demonstrate that the proposed algorithm exhibits remarkable training robustness across DNN models with diverse characteristics. In terms of training convergence, the proposed algorithm demonstrates competitiveness with state-of-the-art algorithms, such as Lion, developed by the Google Brain team.

Similar Papers
  • Conference Article
  • Citations3

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

  • Oct 28, 2023
  • Vahid Janfaza +3
  • Conference Article
  • Citations8

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 01, 2019
  • Bontak Gu +3
  • Research Article
  • Citations179

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

  • May 04, 2021
  • IEEE Transactions on Artificial Intelligence
  • Edgar Galván +1
  • Research Article
  • Citations32

A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networks

  • May 22, 2015
  • Neurocomputing
  • Seyyede Zohreh Seyyedsalehi +1
  • Research Article
  • Citations3

AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks Through Accuracy Gradient

  • Jan 01, 2022
  • IEEE Access
  • Farzad Nikfam +3
  • Research Article
  • Citations30

Selecting and Composing Learning Rate Policies for Deep Neural Networks

  • Feb 16, 2023
  • ACM Transactions on Intelligent Systems and Technology
  • Yanzhao Wu +1
  • Research Article
  • Citations6

A Guessing Entropy-Based Framework for Deep Learning-Assisted Side-Channel Analysis

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Ziyue Zhang +2
  • Research Article
  • Citations1

STADIA: Photonic Stochastic Gradient Descent for Neural Network Accelerators

  • Sep 09, 2023
  • ACM Transactions on Embedded Computing Systems
  • Chengpeng Xia +3
  • Research Article
  • Citations23

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

  • Aug 04, 2023
  • Neurocomputing
  • Jinghui Zhang +6
  • Conference Article
  • Citations132

Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

  • Dec 01, 2019
  • Yanzhao Wu +8
  • Dissertation

A novel model quantization scheme for efficient deep neural networks inference and training

  • Jan 01, 2024
  • Sung-En Chang
  • Conference Article
  • Citations4

MalleTrain: Deep Neural Networks Training on Unfillable Supercomputer Nodes

  • May 07, 2024
  • Xiaolong Ma +6
  • Conference Article
  • Citations11

Training of Deep Neural Networks in Electromagnetic Problems: a Case Study of Antenna Array Pattern Synthesis

  • May 23, 2021
  • 2021 IEEE MTT-S International Wireless Symposium (IWS)
  • Zhao Zhou +7
  • Research Article
  • Citations7

A convergence analysis of Nesterov’s accelerated gradient method in training deep linear neural networks

  • Sep 05, 2022
  • Information Sciences
  • Xin Liu +2
  • Conference Article

Dynamic Look-up Table Method for Optimizing the Training of Deep Neural Networks on Many-Core Architecture

  • Oct 01, 2019
  • Jing Xia +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.