A modified conjugate gradient algorithm for optimization problems

At present, the conjugate gradient (CG) method of Hager and Zhang (Hager and Zhang, SIAM Journal on Optimization, 16(2005)) is regarded as one of the most effective CG methods for optimization problems. In order to further study the CG method, we develop the Hager and Zhang's CG method and present two modified CG formulas, where the given formulas possess the value information of not only the gradient but also the function. Moreover, the sufficient descent condition will be holden without any line search. The global convergence is established for nonconvex function under suitable conditions. Numerical results show that the proposed methods are competitive to the normal conjugate gradient method.

Mathematical programming is a rich and well-developed area in operations research. Nevertheless, there remain many challenging problems in this area, one of which is the large-scale optimization problem. In this article, a modified Hestenes and Stiefel (HS) conjugate gradient (CG) algorithm with a nonmonotone line search technique is presented. This algorithm possesses information about not only the gradient value but also the function value. Moreover, the sufficient descent condition holds without any line search. The global convergence is established for nonconvex functions under suitable conditions. Numerical results show that the proposed algorithm is advantageous to existing CG methods for large-scale optimization problems.

The advantage of the electromagnetic tomography is introduced briefly. Based on conjugate gradient algorithm, modified conjugate gradient algorithm for electromagnetic tomography (EMT) is proposed, which improves quality of reconstructed image and convergence speed efficiently. In the light of the lab electromagnetic tomography system, modified conjugate gradient for reconstructing images is verified. By evaluation of image error and the relevance, regularization, Landweber, conjugate gradient and modified conjugate gradient algorithms are compared. It can draw the conclusion that for different flow models, image error and the correlation using modified conjugate gradient algorithm is superior to the others in lab EMT system.

In this paper, a new inexact line search rule is presented, which is a modified version of the classical Armijo line search rule. With lower cost of computation, a larger descent magnitude of objective function is obtained at every iteration. In addition, the initial step size in the modified line search is adjusted automatically for each iteration. On the basis of this line search, a new cautious Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm is developed. Under some mild assumptions, the global convergence of the algorithm is established for nonconvex optimization problems. Numerical results demonstrate that the proposed method is promising, especially in comparison with the existent methods.

Nonlinear conjugate gradient methods for unconstrained optimization problems are used in many aspects of theoretical and applied sciences. They are iterative methods, so at any iteration a step length is computed using a method called line search. In most cases, the sufficient descent condition plays an important role to prove the global convergence of a conjugate gradient method. Due to its outperformance in practical computation, the Polak-Ribière-Polyak (PRP) conjugate gradient method is widely used for solving nonlinear unconstrained optimization problems. However, the sufficient descent condition of PRP has not established without line search yet. In this study, we established the sufficient descent condition without line search based on the conditions 0 < bkPRP £ x bkFR and 1 < bkPRP £mbkFR, where 0 < x < 1 and m > 1. As a result, we found that under certain conditions, the sufficient descent condition is satisfied when the PRP implemented without line search

An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations

Self-scaled conjugate gradient training algorithms

The proof of the sufficient descent condition of the Wei–Yao–Liu conjugate gradient method under the strong Wolfe–Powell line search

Recently, Zhang (2006) proposed a three‐term modified HS (TTHS) method for unconstrained optimization problems. An attractive property of the TTHS method is that the direction generated by the method is always descent. This property is independent of the line search used. In order to obtain the global convergence of the TTHS method, Zhang proposed a truncated TTHS method. A drawback is that the numerical performance of the truncated TTHS method is not ideal. In this paper, we prove that the TTHS method with standard Armijo line search is globally convergent for uniformly convex problems. Moreover, we propose a new truncated TTHS method. Under suitable conditions, global convergence is obtained for the proposed method. Extensive numerical experiment show that the proposed method is very efficient for the test problems from the CUTE Library.

A generalized data windowing scheme for adaptive conjugate gradient algorithms

The convergence of conjugate gradient method with nonmonotone line search

We present a new solution framework to solve the generalized trust region subproblem (GTRS) of minimizing a quadratic objective over a quadratic constraint. More specifically, we derive a convex quadratic reformulation (CQR) via minimizing a linear objective over two convex quadratic constraints for the GTRS. We show that an optimal solution of the GTRS can be recovered from an optimal solution of the CQR. We further prove that this CQR is equivalent to minimizing the maximum of the two convex quadratic functions derived from the CQR for the case under investigation. Although the latter minimax problem is nonsmooth, it is well structured and convex. We thus develop two steepest descent algorithms corresponding to two different line search rules. We prove global sublinear convergence rates for both algorithms. We also obtain a local linear convergence rate of the first algorithm by estimating the Kurdyka-Łojasiewicz exponent at any optimal solution under mild conditions. We finally demonstrate the efficiency of our algorithms with numerical experiments.

There are two problems for nonconvex functions under the weak Wolfe–Powell line search in unconstrained optimization problems. The first one is the global convergence of the Polak–Ribiere–Polyak conjugate gradient algorithm and the second is the global convergence of the Broyden–Fletcher–Goldfarb–Shanno quasi-Newton method. Many scholars have proven that the two problems do not converge, even under an exact line search. Two circle counterexamples were proposed to generate the nonconvergence of the Polak–Ribiere–Polyak algorithm for the nonconvex functions under the exact line search, which inspired us to define a new technique to jump out of the circle point and obtain the global convergence. Thus, a new Polak–Ribiere–Polyak algorithm is designed by the following steps. (i) Given the current point and a parabolic surface is designed; (ii) An assistant point is defined based on the current point; (iii) The assistant point is projected onto the surface to generate the next point; (iv) The presented algorithm has the global convergence for nonconvex functions with the weak Wolfe–Powell line search. A similar technique is used for the quasi-Newton method to get a new quasi-Newton algorithm and to establish its global convergence. Numerical results show that the given algorithms are more competitive than other similar algorithms. Meanwhile, the well-known hydrologic engineering application problem, called parameter estimation problem of nonlinear Muskingum model, is also done by the proposed algorithms.

A framework for proving global convergence for a class of line search filter-type methods for nonlinear programming is presented without assuming that the Jacobian has full rank everywhere. The underlying method is based on the filter concept where trial points are accepted, provided there is a sufficient decrease in the objective function or constraints violation function. The proposed methods solve a sequence of quadratic programming subproblems via line search techniques to induce global convergence. Under mild conditions, we will also show that the algorithm converges two step superlinearly when the iterates are near to the solution.

Though quasi-Newton methods have been extensively studied in the literature, they either suffer from local convergence or use a series of line searches for global convergence. In this work, we propose a line search free greedy quasi-Newton method with adaptive steps and establish explicit nonasymptotic bounds for both the global convergence rate and local superlinear rate. Our novel idea lies in the design of multiple greedy quasi-Newton updates to control the Hessian approximation error and a simple mechanism to adjust stepsizes to ensure function improvement per iteration. The global superlinear convergence <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> of our method is validated via numerical experiments.