• Home
  • Search
  • A linearly convergent stochastic recursive gradient method for convex optimization
  • Cite Icon15
  • https://doi.org/10.1007/s11590-020-01550-xCopy DOI Icon

A linearly convergent stochastic recursive gradient method for convex optimization

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

The stochastic recursive gradient algorithm (SARAH) (Nguyen et al. in: Neural information processing systems, pp 2613–2621, 2017) attracts much interest recently. It admits a simple recursive framework for updating stochastic gradient estimates. Motivated by this, in this paper we propose a new stochastic recursive gradient method, called SARAH-I. Different from SARAH, SARAH-I incorporates importance sampling strategy and computes the full gradient at the last iterate in each inner iteration. We show that the sequence of distances between iterates and the optima set is linearly convergent under both strong convexity and non-strong convexity conditions. Furthermore, we propose to use the Barzilai–Borwein (BB) approach to adaptively compute step sizes for SARAH-I, and name the resulting method as SARAH-I-BB. We establish its convergence and complexity properties in different cases. Finally numerical tests are reported to indicate promising performances of proposed algorithms.

Similar Papers
  • Research Article
  • Citations31

Biased stochastic conjugate gradient algorithm with adaptive step size for nonconvex problems

  • Sep 22, 2023
  • Expert Systems with Applications
  • Ruping Huang +3
  • Research Article
  • Citations8

Stochastic gradient method with accelerated stochastic dynamics

  • Mar 01, 2016
  • Journal of Physics: Conference Series
  • Masayuki Ohzeki
  • Conference Article
  • Citations7

A Novel Constrained Adaptive Algorithm Using the Conjugate Gradient Method for Smart Antennas

  • Nov 01, 2007
  • Lei Wang +1
  • Research Article
  • Citations1187

Minimizing finite sums with the stochastic average gradient

  • Jun 14, 2016
  • Mathematical Programming
  • Mark Schmidt +2
  • Book Chapter

How Effectively Train Large-Scale Machine Learning Models?

  • Jan 01, 2019
  • Aven Samareh +1
  • Research Article
  • Citations3

Adaptive Biased Stochastic Optimization.

  • Jan 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhuang Yang
  • Research Article

Efficiency of Stochastic Coordinate Proximal Gradient Methods on Nonseparable Composite Optimization

  • Apr 16, 2024
  • Mathematics of Operations Research
  • Ion Necoara +1
  • Research Article
  • Citations3

A comparison of stochastic gradient and minimum entropy deconvolution algorithms

  • Sep 01, 1988
  • Signal Processing
  • A.T Walden
  • Conference Article
  • Citations1

Recursive parameter estimation methods based on gradient search for estimating system parameters of first-order inertial control systems

  • Jul 26, 2021
  • Ling Xu
  • Research Article
  • Citations16

A stochastic conjugate gradient method for the approximation of functions

  • Dec 29, 2011
  • Journal of Computational and Applied Mathematics
  • Hong Jiang +1
  • Research Article
  • Citations32

A Stochastic Proximal Alternating Minimization for Nonsmooth and Nonconvex Optimization

  • Jan 01, 2021
  • SIAM Journal on Imaging Sciences
  • Derek Driggs +4
  • Conference Article

Low complexity variable step size mechanisms for constrained minimum variance CDMA receivers

  • May 17, 2004
  • R.C De Lamare +1
  • Conference Article
  • Citations3

An Optimal Stochastic Compositional Optimization Method with Applications to Meta Learning

  • Jun 06, 2021
  • Yuejiao Sun +2
  • Conference Article
  • Citations1

Improving the Computational Efficiency of Approximate Gradients Using a Multiscale Reservoir Simulation Framework

  • Feb 20, 2017
  • R Moraes +4
  • Research Article
  • Citations55

An alternative to EM for Gaussian mixture models: batch and stochastic Riemannian optimization

  • Mar 19, 2019
  • Mathematical Programming
  • Reshad Hosseini +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.