A three-parameter family of nonlinear conjugate gradient methods

The nonlinear conjugate gradient method (CGM) is a very effective iterative method for solving large-scale optimal problems. In this paper, based on a variant of Polak–Ribiere–Polyak method, two modified CGMs with disturbance factors are proposed. By the disturbance factors, the two proposed methods not only generate sufficient descent direction at each iteration but also converge globally for nonconvex minimization if the strong Wolfe line search is used. Finally, elementary numerical experiment results are reported, which show that the proposed methods are promising.

The gradient-based optimization methods are preferable for the large-scale three-dimensional (3D) magnetotelluric (MT) inverse problem. Compared with the popular nonlinear conjugate gradient (NLCG) method, however, the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) method is less adopted. This paper aims to implement a L-BFGS-based inversion algorithm for the 3D MT problem. And we develop our code on top of the ModEM package, which is highly extensible and popular among the MT community. To accelerate the convergence speed, the preconditioning technique by the affine linear transformation of the original model parameters is used. Two modifications of the conventional L-BFGS algorithm are also made to get a comparable convergence rate with the NLCG method. The impacts of the preconditioner parameters, the regularization parameters, the starting model, etc., on the inversion are evaluated by synthetic examples for both L-BFGS and NLCG methods. And the real MT Kayabe dataset is also inverted by the inversion algorithms. The synthetic tests show that through our L-BFGS inversion algorithm the similar resistivity models can be obtained with that from the NLCG method. For the real data inversion, the L-BFGS method performs more efficiently and reasonable results could be obtained by less iterations of the inversion process than the NLCG method. Thus, we suggest the common usage of the L-BFGS method for the 3D MT inverse problem.

In this chapter, we investigate recently proposed nonlinear conjugate gradient (NCG) methods for shape optimization problems. We briefly introduce the methods as well as the corresponding theoretical background and investigate their performance numerically. The obtained results confirm that the NCG methods are efficient and attractive solution algorithms for shape optimization problems.

A descent nonlinear conjugate gradient method for large-scale unconstrained optimization

Nonlinear conjugate gradient (CG) methods are widely used for solving large scale unconstrained optimization problems. Many studies have been devoted to modified and improve this method. In this paper, a new parameter of CG method that possesses global convergence properties using exact line search is proposed. Numerical results show that the new formula is best and more efficient when compared with the other classical CG methods

We study the development of nonlinear conjugate gradient methods, Fletcher Reeves (FR) and Polak Ribiere (PR). FR extends the linear conjugate gradient method to nonlinear functions by incorporating two changes, for the step length αk a line search is performed and replacing the residual, rk (rk=b-Axk) by the gradient of the nonlinear objective function. The PR method is equivalent to FR method for exact line searches and when the underlying quadratic function is strongly convex. The PR method is basically a variant of FR and primarily differs from it in the choice of the parameter βk. On applying the nonlinear Rosenbrock function to the MATLAB code for the FR and the PR algorithms we observe that the performance of PR method (k=29) is far better than the FR method (k=42). But, we observe that when the MATLAB codes are applied to general nonlinear functions, specifically functions whose minimum is a large negative number not close to zero and the iterates too are large values far off from zero the PR algorithm does not perform well. This problem with the PR method persists even if we run the PR algorithm for more iterations or with an initial guess closer to the actual minimum. To improve the PR algorithm we suggest finding a better weighing parameter βk, using better line search method and/or using specific line search for certain functions and identifying specific restart criteria based on the function to be optimized.

Conjugate gradient methods are a popular class of iterative methods for solving linear systems of equations and nonlinear optimization problems as they do not require the storage of any matrices. In order to obtain a theoretically effective and numerically efficient method, two modified conjugate gradient methods (called the MCB1 and MCB2 methods) are proposed. In which the coefficientβkin the two proposed methods is inspired by the structure of the conjugate gradient parameters in some existing conjugate gradient methods. Under the strong Wolfe line search, the sufficient descent property and global convergence of the MCB1 method are proved. Moreover, the MCB2 method generates a descent direction independently of any line search and produces good convergence properties when the strong Wolfe line search is employed. Preliminary numerical results show that the MCB1 and MCB2 methods are effective and robust in minimizing some unconstrained optimization problems and each of these modifications outperforms the four famous conjugate gradient methods. Furthermore, the proposed algorithms were extended to solve the problem of mode function.

A new nonlinear spectral conjugate descent method for solving unconstrained optimization problems is proposed on the basis of the CD method and the spectral conjugate gradient method. For any line search, the new method satisfies the sufficient descent condition . Moreover, we prove that the new method is globally convergent under the strong Wolfe line search. The numerical results show that the new method is more effective for the given test problems from the CUTE test problem library (Bongartz et al., 1995) in contrast to the famous CD method, FR method, and PRP method.

In this paper, we present a new hybrid method to solve a nonlinear unconstrained optimization problem by using conjugate gradient, which is a convex combination of Liu–Storey (LS) conjugate gradient method and Hager–Zhang (HZ) conjugate gradient method. This method possesses the sufficient descent property with Strong Wolfe line search and the global convergence with the strong Wolfe line search. In the end of this paper, we illustrate our method by giving some numerical examples.

The stationary point of optimization problems can be obtained via conjugate gradient (CG) methods without the second derivative. Many researchers have used this method to solve applications in various fields, such as neural networks and image restoration. In this study, we construct a three-term CG method that fulfills convergence analysis and a descent property. Next, in the second term, we employ a Hestenses-Stiefel CG formula with some restrictions to be positive. The third term includes a negative gradient used as a search direction multiplied by an accelerating expression. We also provide some numerical results collected using a strong Wolfe line search with different sigma values over 166 optimization functions from the CUTEr library. The result shows the proposed approach is far more efficient than alternative prevalent CG methods regarding central processing unit (CPU) time, number of iterations, number of function evaluations, and gradient evaluations. Moreover, we present some applications for the proposed three-term search direction in image restoration, and we compare the results with well-known CG methods with respect to the number of iterations, CPU time, as well as root-mean-square error (RMSE). Finally, we present three applications in regression analysis, image restoration, and electrical engineering.

Recently, similar to Hager and Zhang (SIAM J Optim 16:170–192, 2005), Yu (Nonlinear self-scaling conjugate gradient methods for large-scale optimization problems. Thesis of Doctors Degree, Sun Yat-Sen University, 2007) and Yuan (Optim Lett 3:11–21, 2009) proposed modified PRP conjugate gradient methods which generate sufficient descent directions without any line searches. In order to obtain the global convergence of their algorithms, they need the assumption that the stepsize is bounded away from zero. In this paper, we take a little modification to these methods such that the modified methods retain sufficient descent property. Without requirement of the positive lower bound of the stepsize, we prove that the proposed methods are globally convergent. Some numerical results are also reported.

Purpose– Generally, iterative methods which have some instability solutions in complex structural and non-linear mechanical problems are used to compute reliability index. The purpose of this paper is to establish a non-linear conjugate gradient (NCG) optimization algorithm to overcome instability solution of the Hasofer-Lind and Rackwitz-Fiessler (HL-RF) method in first-order reliability analysis. The NCG algorithms such as the Conjugate-Descent (CD) and the Liu-Storey (LS) are used for determining the safety index. An algorithm is found based on the new line search in the reliability analysis.Design/methodology/approach– In the proposed line search for calculating the safety index, search direction is computed by using the conjugate gradient approach and the HL-RF method based on the new and pervious gradient vector of the reliability function. A simple step size is presented for the line search in the proposed algorithm, which is formulated by the Wolfe conditions based on the new and previous safety index results in the reliability analysis.Findings– From the current work, it is concluded that the proposed NCG algorithm has more efficient, robust and appropriate convergence in comparison with the HL-RF method. The proposed methods can eliminate numerical instabilities of the HL-RF iterative algorithm in highly non-linear performance function and complicated structural limit state function. The NGC optimization is applicable to reliability analysis and it is correctly converged on the reliability index. In the NCG method, the CD algorithm is slightly more efficient than the LS algorithm.Originality/value– This paper usefully shows how the HL-RF algorithm and the NCG scheme are formulated in first-order reliability analysis. The proposed algorithm is validated from six numerical and structural examples taken from the literature. The HL-RF method is not converged on several non-linear mathematic and complex structural examples, while the two proposed conjugate gradient methods are appropriately converged for all examples. The CD algorithm is converged about twice faster than the LS algorithm in most of the problems. Therefore, application of the NCG method is possible in reliability analysis.

Both fully adaptive and partially adaptive processing algorithms can be implemented using linear and nonlinear conjugate gradient (CG) methods. This approach can be computationally attractive when a family of continuous time-varying adaptive problems must be solved. An iterative approach that starts with the previous solution can converge quickly to the new solution. This paper considers linear and nonlinear CG methods for adaptive processing. The example used is space-time adaptive processing (STAP) and clutter interference subspace tracking within the context of an airborne radar with a rotating array. It is seen that the linear CG method for adaptive filtering in this context can be several times less costly than sample matrix inversion if the interference subspace has sufficient eigenvalue structure, and that a nonlinear CG algorithm is capable of tracking the principal clutter interference subspace.

<p>The conjugate gradient methods are noted to be exceedingly valuable for solving large-scale unconstrained optimization problems since it needn't the storage of matrices. Mostly the parameter conjugate is the focus for conjugate gradient methods. The current paper proposes new methods of parameter of conjugate gradient type to solve problems of large-scale unconstrained optimization. A Hessian approximation in a diagonal matrix form on the basis of second and third-order Taylor series expansion was employed in this study. The sufficient descent property for the proposed algorithm are proved. The new method was converged globally. This new algorithm is found to be competitive to the algorithm of fletcher-reeves (FR) in a number of numerical experiments.</p>

A study of Liu-Storey conjugate gradient methods for vector optimization