Convergence rate analysis of the multiplicative gradient method for PET-type problems

The method of steepest descent, or the optimum gradient method, has been known to mathematicians since the time of Cauchy [l]. Others who have discussed this method include Curry 12], Forsythe and Motzkin [3], Householder [4], Kantorovitch [5], and Temple [6]. Its infrequent application in computational work is no doubt due to the slowness with which it converges. This slowness of convergence is unfortunately generally true of gradient methods. However, with the advent of large-scale computing S it has become feasible to seriously consider em in practical numerical analysis. In a forthcoming paper, Hestenes and Stein [7] discuss a large class of gradient procedures for solving systems of linear equations. These procedures contain the optimum gradient method as a special case. The present note is mainly a report on some numerical experiments with them that were carried out on the IBM Card-Programmed Electronic Calculator at the Institute for Numerical Analysis of the National Bureau of Standards. Some attention is also given I to an experiment in which the problem of solving a system of linear equations was changed to an equivalent eigenvalue problem and then solved by a modification of one of the gradient methods discussed by Hestenes and Karush [8]. The most striking result of the experiments indicates that there is a large class of gradient methods that is self-accelerating, that is, which irregularly shows a large increase in the rate of convergence without the introduction of any modification in the computational routine. This behavior is in sharp contrast to that of the method of steepest descent, which in the light of the present results can no longer be considered as optimum from an over-all point of view unless modified by some special accelerating routine [9j.

A great deal of interest of solving large-scale convex optimization problems has, recently, turned to gradient method and its variants. To ensure rates of linear convergence, current theory regularly assumes that the objective functions are strongly convex. This paper goes beyond the traditional wisdom by studying a strictly weaker concept than the strong convexity, named restricted strongly convexity which was recently proposed and can be satisfied by a much broader class of functions. Utilizing the restricted strong convexity, we derive rates of linear convergence for (in)exact gradient-type methods. Besides, we obtain two by-products: (1) we rederive rates of linear convergence of inexact gradient method for a class of structured smooth convex optimizations; (2) we improve the rate of linear convergence for the linearized Bregman algorithm.

This study concerns the fast and accurate solution of the line radiation\ntransfer problem, under non-LTE conditions. We propose and evaluate an\nalternative iterative scheme to the classical ALI-Jacobi method, and to the\nmore recently proposed Gauss-Seidel and Successive Over-Relaxation (GS/SOR)\nschemes. Our study is indeed based on the application of a preconditioned\nbi-conjugate gradient method (BiCG-P). Standard tests, in 1D plane parallel\ngeometry and in the frame of the two-level atom model, with monochromatic\nscattering, are discussed. Rates of convergence between the previously\nmentioned iterative schemes are compared, as well as their respective timing\nproperties. The smoothing capability of the BiCG-P method is also demonstrated.\n

The conjugate gradient (CG) method is considered for solving the large and sparse indefinite least squares (ILS) problem min? x (b?Ax) T J(b?Ax) where J=diag?(I p ,?I q ) is a signature matrix. However the rate of convergence becomes slow for ill-conditioned problems. The QR-based preconditioner is found to be effective in accelerating the convergence. Numerical results show that the sparse Householder QR-based preconditioner is superior to the CG method especially for sparse and ill-conditioned problems.

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from $$O(1/\sqrt{k})$$O(1/k) to O(1 / k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1 / k) to a linear convergence rate of the form $$O(\rho ^k)$$O(?k) for $$\rho < 1$$?<1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

Due to its simplicity and efficiency, the first-order gradient method has been extensively employed in training neural networks. Although the optimization problem of the neural network is non-convex, recent research has proved that the first-order method is capable of attaining a global minimum during training over-parameterized neural networks, where the number of parameters is significantly larger than that of training instances. Momentum methods, including the heavy ball (HB) method and Nesterov's accelerated gradient (NAG) method, are the workhorse of first-order gradient methods owning to their accelerated convergence. In practice, NAG often exhibits superior performance than HB. However, current theoretical works fail to distinguish their convergence difference in training neural networks. To fill this gap, we consider the training problem of the two-layer ReLU neural network under over-parameterization and random initialization. Leveraging high-resolution dynamical systems and neural tangent kernel (NTK) theory, our result not only establishes tighter upper bounds of the convergence rate for both HB and NAG, but also provides the first theoretical guarantee for the acceleration of NAG over HB in training neural networks. Finally, we validate our theoretical results on three benchmark datasets.

In this paper, a new decomposition–coordination framework is presented for two-level optimal control of large-scale nonlinear systems. In the proposed approach, decomposition is performed by defining an interaction vector, while coordination is based on a new interaction prediction approach. In the first level, sub-problems are solved for nonlinear dynamics using a gradient method, while in the second level, the coordination is done using the gradient of coordination errors. This is in contrast to the conventional gradient-type coordination schemes, where they use the gradient of Lagrangian function. It is shown that the proposed decomposition–coordination framework considerably reduces the number of iterations. Moreover, compared to the substitution-type coordination approaches which are well known for their fast convergence rates and limited convergence regions, the proposed approach is robust with respect to parameters’ variation in addition to its good convergence rate. The robustness and the convergence rate of the proposed approach in compare to the costate prediction method are shown through simulations of a benchmark problem. The results are also compared to the ones obtained using centralized approach.

Self-scaled conjugate gradient training algorithms

The normalized least‐mean‐squares (NLMS) method has been adopted for a wide range of adaptive filter applications due to its simple structure and low complexity of computation. However, under certain circumstances, the convergence rate may not be satisfactory. The performance may even be poorer under certain nonstationary situations. A new normalized stochastic gradient (NSG) method is suggested to overcome the drawbacks of the NLMS method. The main features of the NSG method are (1) the time varying convergence parameter and (2) the onset detection. While in the NLMS method the convergence parameter is fixed, it is a function of time in the NSG method. It is selected to minimize the conditional variance of the filter coefficients at the next step given a set of the previous coefficients. The time‐varying sequence of the convergence parameter makes the NSG method converge fast. The new NSG method has the capability of the onset detection. It detects the statistical changes of the desired input and by rese...

The basic principle of the satcom on the move antenna is to detect the error angle between the antenna beam pointing and the satellite direction according to the received satellite signal, and convert it into the corresponding error electrical signal. The controller uses the error electrical signal to control the antenna to make the antenna move in the direction where the error signal decreases. The tracking accuracy of traditional step tracking is not high, and the response speed is very slow, and the signal amplitude fluctuation affects the tracking accuracy. Since the gain graph of the antenna can be regarded as a non-linear stepped objective function, an improved step tracking algorithm based on the gradient method is proposed, which combines the step tracking with the optimization algorithm to achieve precise tracking. The algorithm optimizes the antenna gain pattern as the objective function to find the azimuth and elevation coordinates corresponding to the maximum point of the antenna gain function. The gain function maximization is transformed into the objective function minimization problem, and the minimum value of the objective function is found through the optimization algorithm, which corresponds to the direction of the antenna when the satellite is accurately aligned. The gradient method is selected for optimization, which is stable as a whole and simple iterative process. Theoretical analysis and simulation results show that the algorithm can obtain higher tracking accuracy while maintaining a lower program complexity.

We discuss the choice of the estimation of the optimal solution when random search methods are applied to solve discrete stochastic optimization problems. At the present time, such optimization methods usually estimate the optimal solution using either the feasible solution the method is currently exploring or the feasible solution visited most often so far by the method. We propose using all the observed objective function values generated as the random search method moves around the feasible region seeking an optimal solution to obtain increasingly more precise estimates of the objective function values at the different points in the feasible region. At any given time, the feasible solution that has the best estimated objective function value (largest one for maximization problems; the smallest one for minimization problems) is used as the estimate of the optimal solution. We discuss the advantages of using this approach for estimating the optimal solution and present numerical results showing that modifying an existing random search method to use tnhis approach for estimating the optimal soluation appears to yield improved performance. We also present sereval rate of convergence results for random search methods using our approach for estimating the optimal solution. One these random search methods is a new variant of the stochastic comparison method; in addition to specifying the rate of convergence of this method, we prove that it is guaranteed to converge almost surely to the set of global optimal solutions and present a result that demonstrates that this method is likely to perform well in practice.

Time-domain multi-scale full waveform inversion (FWI) is an important method for seismic imaging. Using different frequency contents of seismic data, this method mitigates the effect of the local-minima problem by sequential inversion. One of the important factors determining the convergence rate of the nonlinear inversion is the precondition of the gradient. Virtual-source precondition has been successfully applied to Laplace-domain and Laplace-Fourier-domain FWI. In this paper, we will examine the application of the virtual-source precondition to time-domain multi-scale FWI. The research results reveal that, compared to the gradient method without precondition, the virtual-source-based gradient method achieves much greater convergence rate while its additional computational cost is very low. Therefore, virtual-source precondition is very effective for time-domain multi-scale FWI. keywords—virtual-source; precondition; full-waveform inversion

Classical iterative methods when applied to the partial solution of the generalized eigenvalue problem Ax =λBx, may yield very poor convergence rates particularly when ill‐conditioned problems are considered. In this paper the preconditioned conjugate gradient (CG) method via the minimization of the Rayleigh quotient and the reverse power method is employed for the partial eigenproblem. The triangular splitting preconditioners employed are obtained from an incomplete Choleski factorization and a partial Evans preconditioner. This approach can dramatically improve the convergence rate of the basic CG method and is applicable to any symmetric eigenproblem in which one of the matrices A, B is positive definite. Because of the renewed interest in CG techniques for FE work on microprocessors and parallel computers, it is believed that this improved approach to the generalized eigenvalue problem is likely to be very promising.

We study the trade-offs between convergence rate and robustness to gradient errors in designing a first-order algorithm. We focus on gradient descent and accelerated gradient (AG) methods for minimizing strongly convex functions when the gradient has random errors in the form of additive white noise. With gradient errors, the function values of the iterates need not converge to the optimal value; hence, we define the robustness of an algorithm to noise as the asymptotic expected suboptimality of the iterate sequence to input noise power. For this robustness measure, we provide exact expressions for the quadratic case using tools from robust control theory and tight upper bounds for the smooth strongly convex case using Lyapunov functions certified through matrix inequalities. We use these characterizations within an optimization problem which selects parameters of each algorithm to achieve a particular trade-off between rate and robustness. Our results show that AG can achieve acceleration while being more robust to random gradient errors. This behavior is quite different than previously reported in the deterministic gradient noise setting. We also establish some connections between the robustness of an algorithm and how quickly it can converge back to the optimal solution if it is perturbed from the optimal point with deterministic noise. Our framework also leads to practical algorithms that can perform better than other state-of-the-art methods in the presence of random gradient noise.

Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown to directly benefit from Nesterov’s acceleration techniques to match the convergence rates of accelerated gradient methods. Such approaches rely on “negative momentum”, a technique for further variance reduction that is generally specific to the SVRG gradient estimator. In this work, we show for the first time that negative momentum is unnecessary for acceleration and develop a universal acceleration framework that allows all popular variance-reduced methods to achieve accelerated convergence rates. The constants appearing in these rates, including their dependence on the number of functions n, scale with the mean-squared-error and bias of the gradient estimator. In a series of numerical experiments, we demonstrate that versions of SAGA, SVRG, SARAH, and SARGE using our framework significantly outperform non-accelerated versions and compare favourably with algorithms using negative momentum.