Stochastic gradient method with accelerated stochastic dynamics

Biased stochastic conjugate gradient algorithm with adaptive step size for nonconvex problems

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from $$O(1/\sqrt{k})$$O(1/k) to O(1 / k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1 / k) to a linear convergence rate of the form $$O(\rho ^k)$$O(?k) for $$\rho < 1$$?<1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

The stochastic gradient method (SGM) is widely used as an optimization tool in many machine learning applications including support vector machines (SVM)s, logistic regression, graphical models and deep learning. SGM computes the estimates of the gradient from a single randomly chosen sample in each iteration. Therefore, applying a stochastic gradient method for large-scale machine learning problems can be computationally efficient. In this work, we focus on generating generalization bounds for a randomized algorithm such as Random Fourier features learned with stochastic gradient descent algorithm. Our findings are based on a mutual relationship between the generalization error of an algorithm and its stability properties. The stability of an algorithm is measured by the generalization error regarding the absolute difference between the testing and the training error. Overall, an algorithm is called stable if by changing any single training data point the training error varies slightly. In this work, we measured the stability of stochastic gradient method (SGM) for learning an approximated Fourier primal support vector machine. In particular, under certain regularity assumptions, we showed that a randomized algorithm such as Random Fourier feature where is trained by a stochastic gradient method (SGM) with few iterations has vanishing generalization error. Therefore, the iterative optimization algorithm can stop long before its convergence to reduce computational cost. We empirically verified the theoretical findings for different parameters using several data sets.

The stochastic recursive gradient algorithm (SARAH) (Nguyen et al. in: Neural information processing systems, pp 2613–2621, 2017) attracts much interest recently. It admits a simple recursive framework for updating stochastic gradient estimates. Motivated by this, in this paper we propose a new stochastic recursive gradient method, called SARAH-I. Different from SARAH, SARAH-I incorporates importance sampling strategy and computes the full gradient at the last iterate in each inner iteration. We show that the sequence of distances between iterates and the optima set is linearly convergent under both strong convexity and non-strong convexity conditions. Furthermore, we propose to use the Barzilai–Borwein (BB) approach to adaptively compute step sizes for SARAH-I, and name the resulting method as SARAH-I-BB. We establish its convergence and complexity properties in different cases. Finally numerical tests are reported to indicate promising performances of proposed algorithms.

This paper deals with composite optimization problems having the objective function formed as the sum of two terms; one has a Lipschitz continuous gradient along random subspaces and may be nonconvex, and the second term is simple and differentiable but possibly nonconvex and nonseparable. Under these settings, we design a stochastic coordinate proximal gradient method that takes into account the nonseparable composite form of the objective function. This algorithm achieves scalability by constructing at each iteration a local approximation model of the whole nonseparable objective function along a random subspace with user-determined dimension. We outline efficient techniques for selecting the random subspace, yielding an implementation that has low cost per iteration, also achieving fast convergence rates. We present a probabilistic worst case complexity analysis for our stochastic coordinate proximal gradient method in convex and nonconvex settings; in particular, we prove high-probability bounds on the number of iterations before a given optimality is achieved. Extensive numerical results also confirm the efficiency of our algorithm. Funding: This work was supported by Norway Grants 2014-2021 [Grant ELO-Hyp 24/2020]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii [Grants PN-III-P4-PCE-2021-0720, L2O-MOC, nr 70/2022]; and the ITN-ETN project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement [Grant 861137].

In this paper, several reported blind adaptive beam- forming algorithms, including the stochastic gradient (SG) methods and the recursive least squares (RLS) methods are reviewed. Then, the conjugate gradient (CG) algorithm is introduced as a tradeoff between SG and RLS algorithms. On the basis of this idea, a novel constrained constant modulus (CCM) design criterion is described and employed for deriving a modified conjugate gradient (CG) optimization algorithm, which does not require the information of the desired user beforehand. Simulations are performed to compare the performance of the proposed method with both SG and RLS methods. The results indicate that the proposed approach has a significant faster convergence rate and superior resolution compared with the SG algorithms and lower computation complexity compared with the RLS algorithms.

A stochastic conjugate gradient method for the approximation of functions

Recognizing the central role of Lagrange multipliers in network resource allocation, a novel learn-and-adapt stochastic dual gradient (LA-SDG) method is developed in this paper to learn the empirical optimal Lagrange multiplier from historical data, and adapt to the upcoming resource allocation strategy. Remarkably, it only requires one more sample (gradient) evaluation than the celebrated stochastic dual gradient (SDG) method. LA-SDG can be interpreted as a foresighted learning approach with an eye on the future. It is established - both theoretically and empirically - that LA-SDG markedly improves the cost-delay tradeoff over state-of-the-art network resource allocation schemes.

We present an efficient workflow that combines multiscale (MS) forward simulation and stochastic gradient computation - MS-StoSAG - for the optimization of well controls applied to waterflooding under geological uncertainty. The Iterative Multiscale Finite Volume (i-MSFV), a mass conservative reservoir simulation strategy, is employed as the forward simulation strategy. The i-MSFV method has the ability to accurately capture fine scale heterogeneities, and thus the fine scale physics of the problem. This allows for accurate and efficient simulations with fine-scale error estimates in a more computationally efficient coarse-scale simulation grid. The Stochastic Simplex Approximate Gradient (StoSAG) method, a stochastic gradient technique, inspired from Ensemble Optimization (EnOpt), is employed to compute the gradient of the objective function. Stochastic methods have recently been shown to be very efficient in terms of gradient quality and computational cost especially for robust optimization problems. They rely on the reservoir simulator as a black-box and perform a large number of forward simulations to approximate a gradient. In our numerical experiments, we investigate the impact of MS parameters such as coarsening ratio and heterogeneity contrast on the quality of the approximate MS-StoSAG gradient. Our experiments illustrate that i-MSFV simulations provide accurate forward simulation responses for the gradient computation, with the advantage of speeding up the workflow due to faster simulations. Speedups up to a factor of five on the forward simulation were achieved for the examples considered in the paper. The combination of speed and accuracy of MS forward simulation with the flexibility of the StoSAG technique (several different parameters/responses can be easily considered) allows for a flexible and efficient optimization strategy suitable for large-scale problems.

This paper studies the recursive parameter estimation methods for the first-order inertial system based on the impulse responses and the gradient search. By constructing a cost function with respect to system parameters in accordance with the output response and observed data, the least mean squares method, the stochastic method and the multi-innovation method are proposed. The stochastic gradient method is presented to avoid solving complicated one-dimensional equation for determining step-sizes and the multi-innovation stochastic is proposed to enhance the estimation accuracy. The simulation results are provided and show that the proposed methods are effective.

Stochastic compositional optimization generalizes classic (non-compositional) stochastic optimization to the minimization of com-positions of functions. Each composition may introduce an additional expectation. The series of expectations may be nested. Stochastic compositional optimization is gaining popularity in applications such as meta learning. This paper presents a new Stochastically Corrected Stochastic Compositional gradient method (SCSC). SCSC runs in a single-time scale with a single loop, uses a fixed batch size, and guarantees to converge at the same rate as the stochastic gradient descent (SGD) method for non-compositional stochastic optimization. It is easy to apply SGD-improvement techniques to accelerate SCSC. This helps SCSC achieve state-of-the-art performance for stochastic compositional optimization. In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional optimization. We test SCSC using the model-agnostic meta-learning tasks.

In network semi-supervised learning problems, only a subset of the network nodes is able to access the data labeling. This paper formulates a decentralized optimization problem where agents have individual decision rules to estimate, subject to the condition that neighboring agents (classifiers) are more likely to have similar labels. To promote such relationships, we propose to add to the aggregate sum of individual costs a graph regularization term that allows to penalize the differences between the labels at neighboring agents. Streaming data is assumed, and therefore, the stochastic (sub-)gradient method is used to solve the regularized problem. We provide conditions that guarantee the stability and convergence of the proposed algorithm. Simulation results show that collaboration among neighboring agents leads to better classification results by decreasing the probability of error and by improving the convergence rate.

The normalized least‐mean‐squares (NLMS) method has been adopted for a wide range of adaptive filter applications due to its simple structure and low complexity of computation. However, under certain circumstances, the convergence rate may not be satisfactory. The performance may even be poorer under certain nonstationary situations. A new normalized stochastic gradient (NSG) method is suggested to overcome the drawbacks of the NLMS method. The main features of the NSG method are (1) the time varying convergence parameter and (2) the onset detection. While in the NLMS method the convergence parameter is fixed, it is a function of time in the NSG method. It is selected to minimize the conditional variance of the filter coefficients at the next step given a set of the previous coefficients. The time‐varying sequence of the convergence parameter makes the NSG method converge fast. The new NSG method has the capability of the onset detection. It detects the statistical changes of the desired input and by rese...

Adaptive control of systems with uncertainty can be implemented without complete identification of the plant if the cost function is measured continuously and the parameters of the controller are adjusted, using an efficient gradient method, to obtain optimum performance. As, in practice, only noisy measurements of the cost function are available, one has to use stochastic gradient methods. In this paper some efficient stochastic approximation algorithms are proposed for adaptive control.

In this work, we generalized and unified two recent completely different works of Jascha [10] and Lee [2] respectively into one by proposing the proximal stochastic Newton-type gradient (PROXTONE) method for optimizing the sums of two convex functions: one is the average of a huge number of smooth convex functions, and the other is a nonsmooth convex function. Our PROXTONE incorporates second order information to obtain stronger convergence results, that it achieves a linear convergence rate not only in the value of the objective function, but also for the solution. The proofs are simple and intuitive, and the results and technique can be served as a initiate for the research on the proximal stochastic methods that employ second order information. The methods and principles proposed in this paper can be used to do logistic regression, training of deep neural network and so on. Our numerical experiments shows that the PROXTONE achieves better computation performance than existing methods.