Global convergence of conjugate gradient method

Armijo method is a kind of line search method that usually be used when look for the step size in nonlinear optimization. This paper makes the summary of its modified forms, and then the nonmonotone Armijo-type line search methods are proposed. The numerical results will show that some line search methods with the novel nonmonotone line search are available and efficient in practical computation.

The convergence of conjugate gradient method with nonmonotone line search

In this paper, we present two nonmonotone versions of adaptive cubic regularized (ARC) method for unconstrained optimization problems. The proposed methods are a combination of the ARC algorithm with the nonmonotone line search methods introduced by Zhang and Hager [A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14 (2004), pp. 1043–1056] and Ahookhosh et al. [A nonmonotone trust-region line search method for large-scale unconstrained optimization, Appl. Math. Model. 36 (2012), pp. 478–487]. The global convergence analysis for these iterative algorithms is established under suitable conditions. Several numerical examples are given to illustrate the efficiency and robustness of the newly suggested methods. The obtained results show the satisfactory performance of the proposed algorithms when compared to the basic ARC algorithm.

We study the rate of convergence of a projected Barzilai–Borwein method, which performs the Grippo–Lampariello–Lucidi (GLL) non-monotone line search along the feasible direction, for convex constrained optimization. Under mild conditions, we establish sublinear convergence of the considered method for general convex functions. Moreover, we show that the rate of convergence is R-linear for strongly convex functions. We extend the convergence results to different versions of the method based on a general GLL non-monotone line search, an adaptive non-monotone line search, and a non-monotone line search that requires an average of the successive function values decreases.

In this article, we propose a new conjugate gradient type formula for computing unconstrained optimization problems. Its form is similar to the original PRP formula and it inherits all nice properties of the PRP method. By utilizing the technique of the three-term PRP method in Zhang [20] and modified PRP method in Yu [17], we propose two modified methods of the new formula. The two modified methods all can generate sufficient descent directions which is independent of the line search used. Under some mild conditions, the global convergence and the linearly convergent rate of the two modified methods are established. The numerical results show that the proposed methods are efficient.

A new nonlinear spectral conjugate descent method for solving unconstrained optimization problems is proposed on the basis of the CD method and the spectral conjugate gradient method. For any line search, the new method satisfies the sufficient descent condition . Moreover, we prove that the new method is globally convergent under the strong Wolfe line search. The numerical results show that the new method is more effective for the given test problems from the CUTE test problem library (Bongartz et al., 1995) in contrast to the famous CD method, FR method, and PRP method.

In this study, we study the application of a kind of nonmonotone line search in BFGS algorithm for solving unconstrained optimization problems. This nonmonotone line search is belongs to Armijo-type line searches and when the step size is being computed at each iteration, the initial test step size can be adjusted according to the characteristics of objective functions. The global convergence of the algorithm is proved. Experiments on some well-known optimization test problems are presented to show the robustness and efficiency of the proposed algorithms.

A sequential quadratically constrained quadratic programming method with an augmented Lagrangian line search function

Static optimization with linear equality constraints and separable structures is studied by using the mixed coordination method. The idea is to relax equality constraints via Lagrange multipliers, and create a hierarchy where the Lagrange multipliers and part of the decision variables are selected as high-level variables. The method was proposed about ten years ago with a simple high-level updating scheme. We show that the solution of the high-level problem is a saddle point, and the simple updating scheme has a linear convergence rate under appropriate conditions. To obtain faster convergence, the modified Newton method is adopted at the high level. There are two difficulties associated with this approach. One is how to obtain the hessian matrix in determining the Newton direction, since second-order derivatives of the objective function with respect to all high-level variables are needed. The second is when to stop in performing a line search along the Newton direction, as the high-level problem is a maxmini problem looking for a saddle point. In this paper, the hessian matrix is obtained by using a kind of sensitivity analysis. The line search stopping criterion, on the other hand, is based on the norm of the gradient vector. Extensive numerical testings show that our approach performs much better than the simple high-level updating scheme. Since the low level consists of a set of independent subproblems, this method is well suited for parallel implementation in solving large-scale problems. Simulated parallel-processing results show that our method outperforms the one-level Lagrange relaxation method for all the test problems. Furthermore, since convexification terms can be added while maintaining the separability of low-level subproblems, the method is very promising for non-convex problems.

This paper proposes a novel nonmonotone proximal gradient method for solving nondifferentiable composite multiobjective optimization problems. The method integrates a nonmonotone line search technique into the proximal gradient framework, offering enhanced flexibility and efficiency for handling nonsmooth or nonconvex objective functions. We analyze the convergence properties of the proposed algorithm, demonstrate that all accumulation points of the generated sequence satisfy Pareto stationarity when the problem is nonconvex. Furthermore, under convexity and mild conditions, we establish that the sequence converges to a weakly Pareto optimal solution of the problem. Numerical experiments on robust multiobjective optimization problems and Lasso multiobjective optimization problems are provided to illustrate the computational efficiency and the practical utility of the proposed method.

In this paper, a filter QP-free infeasible method with nonmonotone line search is proposed for minimizing a smooth optimization problem with smooth inequality constraints. This proposed method is based on the solution of nonsmooth equations, which are obtained by the Lagrangian multiplier method and the function of the nonlinear complementarity problem for the Karush–Kuhn–Tucker optimality conditions. Especially, each iteration of this method can be viewed as a perturbation of a Newton or quasi-Newton iteration on both the primal and dual variables for the solution of the Karush–Kuhn–Tucker optimality conditions. What is more, it is considered to use the function of the nonlinear complementarity problem in the filter, which makes the proposed algorithm avoid the incompatibility. Then the global convergence of the proposed method is given. And under some mild conditions, the superlinear convergence rate can be obtained. Finally, some preliminary numerical results are shown to illustrate that the proposed filter QP-free infeasible method is quite promising.

Motivated by the theoretical and practical results in compressed sensing, efforts have been undertaken by the inverse problems community to derive analogous results, for instance linear convergence rates, for Tikhonov regularization with l1-penalty term for the solution of ill-posed equations. Conceptually, the main difference between these two fields is that regularization in general is an uncon strained optimization problem, while in compressed sensing a constrained one is used. Since the two methods have been developed in two different communities, the theoretical approaches to them appear to be rather different: In compressed sensing, the restricted isometry property seems to be central for proving linear convergence rates, whereas in regularization theory range or source conditions are imposed. The paper gives a common meaning to the seemingly different conditions and puts them into perspective with the conditions from the respective other community. A particularly important observation is that the range condition together with an injectivity condition is weaker than the restricted isometry property. Under the weaker conditions, linear convergence rates can be proven for compressed sensing and for Tikhonov regularization. Thus existing results from the literature can be improved based on a unified analysis. In particular, the range condition is shown to be the weakest possible condition that permits the derivation of linear convergence rates for Tikhonov regularization with a priori parameter choice. © 2010 Wiley Periodicals, Inc.

In this paper, a new nonmonotone adaptive trust region algorithm is proposed for unconstrained optimization by combining a multidimensional filter and the Goldstein-type line search technique. A modified trust region ratio is presented which results in more reasonable consistency between the accurate model and the approximate model. When a trial step is rejected, we use a multidimensional filter to increase the likelihood that the trial step is accepted. If the trial step is still not successful with the filter, a nonmonotone Goldstein-type line search is used in the direction of the rejected trial step. The approximation of the Hessian matrix is updated by the modified Quasi-Newton formula (CBFGS). Under appropriate conditions, the proposed algorithm is globally convergent and superlinearly convergent. The new algorithm shows better performance in terms of the Dolan–Moré performance profile. Numerical results demonstrate the efficiency and robustness of the proposed algorithm for solving unconstrained optimization problems.

Two modified HS type conjugate gradient methods for unconstrained optimization problems

Nonlinear conjugate gradient methods for unconstrained optimization problems are used in many aspects of theoretical and applied sciences. They are iterative methods, so at any iteration a step length is computed using a method called line search. In most cases, the sufficient descent condition plays an important role to prove the global convergence of a conjugate gradient method. Due to its outperformance in practical computation, the Polak-Ribière-Polyak (PRP) conjugate gradient method is widely used for solving nonlinear unconstrained optimization problems. However, the sufficient descent condition of PRP has not established without line search yet. In this study, we established the sufficient descent condition without line search based on the conditions 0 < bkPRP £ x bkFR and 1 < bkPRP £mbkFR, where 0 < x < 1 and m > 1. As a result, we found that under certain conditions, the sufficient descent condition is satisfied when the PRP implemented without line search