An efficient gradient method with approximate optimal stepsize for large-scale unconstrained optimization

The Barzilai-Borwein (BB) method is a popular and efficient tool for solving large-scale unconstrained optimization problems.Its search direction is the same as for the steepest descent (Cauchy) method, but its stepsize rule is different.Owing to this, it converges much faster than the Cauchy method.A feature of the BB method is that it may generate too long steps, which throw the iterates too far away from the solution.Moreover, it may not converge, even when the objective function is strongly convex.In this paper, a stabilization technique is introduced.It consists in bounding the distance between each pair of successive iterates, which often allows for decreasing the number of BB iterations.When the BB method does not converge, our simple modification of this method makes it convergent.For strongly convex functions with Lipschits gradients, we prove its global convergence, despite the fact that no line search is involved, and only gradient values are used.Since the number of stabilization steps is proved to be finite, the stabilized version inherits the fast local convergence of the BB method.The presented results of extensive numerical experiments show that our stabilization technique often allows the BB method to solve problems in a fewer iterations, or even to solve problems where the latter fails.

In this paper, a three-dimensional subspace conjugate gradient method is proposed, in which the search direction is generated by minimizing the approximation model of the objective function in a three-dimensional subspace. The approximation model is not unique and is alternative between quadratic model and conic model by the specific criterions. The strategy of initial stepsize and nonmonotone line search are adopted, and the global convergence of the presented algorithm is established under mild assumptions. In numerical experiments, we use a collection of 80 unconstrained optimization test problems to show the competitive performance of the presented method.

The adaptive cyclic Barzilai-Borwein (BB) method [DZ05] for unconstrained optimization is extended to bound constrained optimization. Using test problems from the CUTE library [BCGT95], performance is compared with SPG2 (a BB method), GENCAN (a BB/conjugate gradient scheme), and L-BFGS-B (limited BFGS for bound constrained problems).

In this study, we want to develop an efficient optimization algorithm based on Perturbation Analysis (PA) and compare the performance of PA with the traditional Finite Difference (FD) gradient method. While the FD estimate is a very time‐consuming gradient method, PA is a very promising algorithm in terms of the computational time. This is the major advantage of PA over FD. In this study, we use the property that PA only requires one simulation run to obtain other advantages of PA as a gradient method. First, we deal with the problem of initialization bias for the estimate of gradients of system performance. Second, we compare the efficiency of PA and FFD (which represents FD) in introducing the control variates variance reduction technique. Third, we propose a new method for determining the simulation run length at each iteration in an optimization algorithm. In the above three areas, we show that PA is better than FD. Finally, we combine PA and BM (batch means method) to obtain a more efficient gradient method, PA‐BM.

Conjugate gradient methods are a class of very effective iterative methods for large-scale unconstrained optimization. In this paper, a new Dai-Liao conjugate gradient method for solving large-scale unconstrained optimization problem is proposed. Based on the approximately optimal stepsize for the gradient method, we derive three new choices for the important parameters tk in Dai-Liao conjugate gradient method. The search direction satisfies the sufficient descent condition, and the global convergences of the proposed method for uniformly convex and general functions are proved under some mild conditions. Numerical experiments on a set of test problems from the CUTEst library show that the proposed method is superior to some well-known conjugate gradient methods.

Adaptive control of systems with uncertainty can be implemented without complete identification of the plant if the cost function is measured continuously and the parameters of the controller are adjusted, using an efficient gradient method, to obtain optimum performance. As, in practice, only noisy measurements of the cost function are available, one has to use stochastic gradient methods. In this paper some efficient stochastic approximation algorithms are proposed for adaptive control.

A variant of trust-region methods for unconstrained optimization

Self-scaled conjugate gradient training algorithms

<abstract><p>The conjugate gradient (CG) method is an optimization method, which, in its application, has a fast convergence. Until now, many CG methods have been developed to improve computational performance and have been applied to real-world problems. In this paper, a new hybrid three-term CG method is proposed for solving unconstrained optimization problems. The search direction is a three-term hybrid form of the Hestenes-Stiefel (HS) and the Polak-Ribiére-Polyak (PRP) CG coefficients, and it satisfies the sufficient descent condition. In addition, the global convergence properties of the proposed method will also be proved under the weak Wolfe line search. By using several test functions, numerical results show that the proposed method is most efficient compared to some of the existing methods. In addition, the proposed method is used in practical application problems for image restoration and portfolio selection.</p></abstract>

In this paper, we present two nonmonotone versions of adaptive cubic regularized (ARC) method for unconstrained optimization problems. The proposed methods are a combination of the ARC algorithm with the nonmonotone line search methods introduced by Zhang and Hager [A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14 (2004), pp. 1043–1056] and Ahookhosh et al. [A nonmonotone trust-region line search method for large-scale unconstrained optimization, Appl. Math. Model. 36 (2012), pp. 478–487]. The global convergence analysis for these iterative algorithms is established under suitable conditions. Several numerical examples are given to illustrate the efficiency and robustness of the newly suggested methods. The obtained results show the satisfactory performance of the proposed algorithms when compared to the basic ARC algorithm.

Conjugate gradient (CG) methods are renowned for their efficiency in optimizing problems due to their speed and memory use. However, certain formulations of CG methods frequently change between different parameters, potentially omitting valuable values. Additionally, proving adequate descent properties in these methods often relies on a strategy that assumes the ability to exclude specific function values. This study introduces a novel spectral CG (SCG) algorithm that integrates both the spectral gradient parameter and conjugate gradient coefficient. This method stands out for its applicability to large-scale unconstrained optimization problems. Initial numerical findings employing a robust Wolfe line search are provided to showcase the method's efficiency. The spectral coefficient in the field of optimization is a concept used to enhance the performance of search algorithms in unconstrained optimization. It is typically employed in the context of line search methods to improve the convergence rate of algorithms. In optimization algorithms, the spectral coefficient helps adjust the step size to achieve faster convergence towards the optimal solution. The coefficient is calculated based on information from previous iterations of the algorithm. A common application of this concept is in the Spectral Gradient method, where the spectral coefficient is used to modify the gradient direction to accelerate the search process. In this method, the spectral coefficient is used to determine the step length in each iteration, aiming to achieve an optimal balance between speed and stability in finding the optimal solution

This paper reports two proposals of possible preconditioners for the Nonlinear Conjugate Gradient (NCG) method, in large scale unconstrained optimization. On one hand, the common idea of our preconditioners is inspired to L-BFGS quasi–Newton updates, on the other hand we aim at explicitly approximating in some sense the inverse of the Hessian matrix. Since we deal with large scale optimization problems, we propose matrix–free approaches where the preconditioners are built using symmetric low–rank updating formulae. Our distinctive new contributions rely on using information on the objective function collected as by-product of the NCG, at previous iterations. Broadly speaking, our first approach exploits the secant equation, in order to impose interpolation conditions on the objective function. In the second proposal we adopt and ad hoc modified–secant approach, in order to possibly guarantee some additional theoretical properties.

The basic principle of the satcom on the move antenna is to detect the error angle between the antenna beam pointing and the satellite direction according to the received satellite signal, and convert it into the corresponding error electrical signal. The controller uses the error electrical signal to control the antenna to make the antenna move in the direction where the error signal decreases. The tracking accuracy of traditional step tracking is not high, and the response speed is very slow, and the signal amplitude fluctuation affects the tracking accuracy. Since the gain graph of the antenna can be regarded as a non-linear stepped objective function, an improved step tracking algorithm based on the gradient method is proposed, which combines the step tracking with the optimization algorithm to achieve precise tracking. The algorithm optimizes the antenna gain pattern as the objective function to find the azimuth and elevation coordinates corresponding to the maximum point of the antenna gain function. The gain function maximization is transformed into the objective function minimization problem, and the minimum value of the objective function is found through the optimization algorithm, which corresponds to the direction of the antenna when the satellite is accurately aligned. The gradient method is selected for optimization, which is stable as a whole and simple iterative process. Theoretical analysis and simulation results show that the algorithm can obtain higher tracking accuracy while maintaining a lower program complexity.

Low-rank matrix factorization problems such as non negative matrix factorization (NMF) can be categorized as a clustering or dimension reduction technique. The latter denotes techniques designed to find representations of some high dimensional dataset in a lower dimensional manifold without a significant loss of information. If such a representation exists, the features ought to contain the most relevant features of the dataset. Many linear dimensionality reduction techniques can be formulated as a matrix factorization. In this paper, we combine the conjugate gradient (CG) method with the Barzilai and Borwein (BB) gradient method, and propose a BB scaling CG method for NMF problems. The new method does not require to compute and store matrices associated with Hessian of the objective functions. Moreover, adopting a suitable BB step size along with a proper nonmonotone strategy which comes by the size of convex parameter ηk, results in a new algorithm that can significantly improve the CPU time, efficiency, the number of function evaluation. Convergence result is established and numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient in comparison with existing methods and demonstrate the superiority of our algorithms.

Movtivated by the idea of Barzilai and Borwein (BB) method, we introduce a concept of approximately optimal stepsize and present a new gradient method with approximately optimal stepsize for unconstrained optimization. We construct a new quadratic approximation model to generate an approximately optimal stepsize. We then use the two BB stepsizes to truncate it and treat the resulted approximately optimal stepsize as the new stepsize for gradient method. Moreover, for the nonconvex case, we also design two approximate models to generate approximately optimal stepsizes for gradient method. We analyze the convergence of the proposed method. The numerical results show that the proposed method is very promising.