• Home
  • Search
  • ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction
  • Cite Icon3
  • https://doi.org/10.1145/3613424.3623779Copy DOI Icon

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

  • Oct 28, 2023
  • Vahid Janfaza +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Neural network training is inherently sequential where the layers finish the forward propagation in succession, followed by the calculation and back-propagation of gradients (based on a loss function) starting from the last layer. The sequential computations significantly slow down neural network training, especially the deeper ones. Prediction has been successfully used in many areas of computer architecture to speed up sequential processing. Therefore, we propose ADA-GP, which uses gradient prediction adaptively to speed up deep neural network (DNN) training while maintaining accuracy. ADA-GP works by incorporating a small neural network to predict gradients for different layers of a DNN model. ADA-GP uses a novel tensor reorganization method to make it feasible to predict a large number of gradients. ADA-GP alternates between DNN training using backpropagated gradients and DNN training using predicted gradients. ADA-GP adaptively adjusts when and for how long gradient prediction is used to strike a balance between accuracy and performance. Last but not least, we provide a detailed hardware extension in a typical DNN accelerator to realize the speed up potential from gradient prediction. Our extensive experiments with fifteen DNN models show that ADA-GP can achieve an average speed up of 1.47 × with similar or even higher accuracy than the baseline models. Moreover, it consumes, on average, 34% less energy due to reduced off-chip memory accesses compared to the baseline accelerator.

Similar Papers
  • Research Article
  • Citations179

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

  • May 04, 2021
  • IEEE Transactions on Artificial Intelligence
  • Edgar Galván +1
  • Conference Article
  • Citations8

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 01, 2019
  • Bontak Gu +3
  • Research Article
  • Citations6

A Guessing Entropy-Based Framework for Deep Learning-Assisted Side-Channel Analysis

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Ziyue Zhang +2
  • Conference Article
  • Citations4

MalleTrain: Deep Neural Networks Training on Unfillable Supercomputer Nodes

  • May 07, 2024
  • Xiaolong Ma +6
  • PDF
  • Research Article
  • Citations13

Identification of Intrinsically Disordered Protein Regions Based on Deep Neural Network-VGG16

  • Mar 28, 2021
  • Algorithms
  • Pengchang Xu +2
  • Research Article
  • Citations2

Accelerating deep network training for radar identification using batch normalization

  • Nov 25, 2024
  • Defence Science Journal
  • Preeti Gupta +2
  • Conference Article
  • Citations7

Comparison of Loss Functions for Training of Deep Neural Networks in Shogi

  • Nov 01, 2018
  • Hanhua Zhu +1
  • Conference Article
  • Citations11

Training of Deep Neural Networks in Electromagnetic Problems: a Case Study of Antenna Array Pattern Synthesis

  • May 23, 2021
  • 2021 IEEE MTT-S International Wireless Symposium (IWS)
  • Zhao Zhou +7
  • Research Article

SVRG-AALR: Stochastic Variance-Reduced Gradient Method with Adaptive Alternating Learning Rate for Training Deep Neural Networks

  • Jul 25, 2025
  • Electronics
  • Shiyun Zou +3
  • Research Article
  • Citations23

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

  • Aug 04, 2023
  • Neurocomputing
  • Jinghui Zhang +6
  • Research Article
  • Citations10

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

  • Aug 01, 2023
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Qing Ye +4
  • Research Article
  • Citations11

Low Complexity Gradient Computation Techniques to Accelerate Deep Neural Network Training.

  • Sep 01, 2023
  • IEEE Transactions on Neural Networks and Learning Systems
  • Dongyeob Shin +3
  • Conference Article
  • Citations3

AccDP: Accelerated Data-Parallel Distributed DNN Training for Modern GPU-Based HPC Clusters

  • Dec 01, 2022
  • Nawras Alnaasan +4
  • Conference Article
  • Citations30

Investigations on hessian-free optimization for cross-entropy training of deep neural networks

  • Aug 25, 2013
  • Simon Wiesler +2
  • Research Article
  • Citations52

TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems

  • Mar 31, 2021
  • IEEE Transactions on Very Large Scale Integration (VLSI) Systems
  • Sourjya Roy +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.