A three-term CG projected-based method with control factor and its inertial version for constrained nonlinear equations

A matrix-free method for constrained equations is proposed, which is a combination of the well-known PRP (Polak-Ribière-Polyak) conjugate gradient method and the famous hyperplane projection method. The new method is not only derivative-free, but also completely matrix-free, and consequently, it can be applied to solve large-scale constrained equations. We obtain global convergence of the new method without any differentiability requirement on the constrained equations. Compared with the existing gradient methods for solving such problem, the new method possesses linear convergence rate under standard conditions, and a relax factorγis attached in the update step to accelerate convergence. Preliminary numerical results show that it is promising in practice.

Motivated by the theoretical and practical results in compressed sensing, efforts have been undertaken by the inverse problems community to derive analogous results, for instance linear convergence rates, for Tikhonov regularization with l1-penalty term for the solution of ill-posed equations. Conceptually, the main difference between these two fields is that regularization in general is an uncon strained optimization problem, while in compressed sensing a constrained one is used. Since the two methods have been developed in two different communities, the theoretical approaches to them appear to be rather different: In compressed sensing, the restricted isometry property seems to be central for proving linear convergence rates, whereas in regularization theory range or source conditions are imposed. The paper gives a common meaning to the seemingly different conditions and puts them into perspective with the conditions from the respective other community. A particularly important observation is that the range condition together with an injectivity condition is weaker than the restricted isometry property. Under the weaker conditions, linear convergence rates can be proven for compressed sensing and for Tikhonov regularization. Thus existing results from the literature can be improved based on a unified analysis. In particular, the range condition is shown to be the weakest possible condition that permits the derivation of linear convergence rates for Tikhonov regularization with a priori parameter choice. © 2010 Wiley Periodicals, Inc.

An adaptive conjugate gradient algorithm for large-scale unconstrained optimization

Matrix completion refers to recovering a matrix from a small subset of its entries. It is an important topic because numerous real-world data can be modeled as low-rank matrices. One popular approach for matrix completion is based on low-rank matrix factorization, but it requires knowing the matrix rank, which is difficult to accurately determine in many practical scenarios. We propose a novel algorithm based on rank-one approximation that a matrix can be decomposed as a sum of outer products. The key idea is to find the basis vectors of the underlying matrix according to the observed entries, and gradually increase the vector number until an appropriate rank estimate is reached. In contrast to the conventional rank-one schemes that employ unchanging rank-one basis matrices, our algorithm performs completion from the vector viewpoint and is able to generate continuously updated rank-one basis matrices. Besides, we theoretically show that the developed method has a linear convergence rate and a smaller recovery error than existing rank-one based algorithms. Experimental results using both synthetic data and real-world images demonstrate that our solution has the best recovery performance among the competing algorithms when the observations are contaminated by Gaussian noise.

In this paper, we consider the convergence rate of the alternating direction method of multipliers for solving the nonconvex separable optimization problems. Based on the error bound condition, we prove that the sequence generated by the alternating direction method of multipliers converges locally to a critical point of the nonconvex optimization problem in a linear convergence rate, and the corresponding sequence of the augmented Lagrangian function value converges in a linear convergence rate. We illustrate the analysis by applying the alternating direction method of multipliers to solving the nonconvex quadratic programming problems with simplex constraint, and comparing it with some state-of-the-art algorithms, the proximal gradient algorithm, the proximal gradient algorithm with extrapolation, and the fast iterative shrinkage–thresholding algorithm.

The problem of solving discrete-time Lyapunov equations (DTLEs) is investigated over multiagent network systems, where each agent has access to its local information and communicates with its neighbors. To obtain a solution to DTLE, a distributed algorithm with uncoordinated constant step sizes is proposed over time-varying topologies. The convergence properties and the range of constant step sizes of the proposed algorithm are analyzed. Moreover, a linear convergence rate is proved and the convergence performances over dynamic networks are verified by numerical simulations.

A spectral conjugate gradient method for solving large-scale unconstrained optimization

In this paper, we present two families of modified three-term conjugate gradient methods for solving unconstrained large-scale smooth optimization problems. We show that our new families satisfy the Dai-Liao conjugacy condition and the sufficient descent condition under any line search technique which guarantees the positiveness of ${y_{k}^{T}} s_{k}$. For uniformly convex functions, we indicate that our families are globally convergent under weak-Wolfe-Powell line search technique and standard conditions on the objective function. We also establish a weaker global convergence theorem for general smooth functions under similar assumptions. Our numerical experiments for 260 standard problems and seven other recently developed conjugate gradient methods illustrate that the members of our families are numerically efficient and effective.

This study introduces an improved three-term conjugate gradient (CG) algorithm which fulfills the descent conditions and exhibits good global convergence properties. The development of this new algorithm is based on the findings of recently introduced generalized RMIL CG technique. The algorithm has been modified so that the search direction would always meet the sufficient descent condition no matter the line-search techniques employed. Under some given assumptions, the algorithm’s global convergence result for the general nonconvex functions is established. The numerical efficacy of the suggested three-term CG algorithm is assessed through comparisons with four other CG algorithms. The assessment is performed using an array of unconstrained optimization benchmark functions. The acquired findings indicate that all the considered algorithms exhibit similar performance, particularly the variants of the RMIL method. One algorithm is more robust and slightly faster than the rest. However, it is essential to highlight that the newly developed CG formula outperforms all the other algorithms, including a recently presented three-term PRP algorithm. The study further expanded the proposed method to address an image restoration problem. The computational experiments yielded promising results, proving that the proposed CG algorithm surpasses others. It produces higher-quality output images, requires less CPU time (sec), and achieves higher PSNR values.

In this paper, a novel root finding problem for the Lagrangian support vector regression in 2-norm (LSVR) is formulated in which the number of unknowns becomes the number of training examples. Further, it is proposed to solve it by functional iterative and Newton methods. Under sufficient conditions, we proved their linear rate of convergence. Experiments are performed on a number of synthetic and real-world benchmark datasets, and their results are compared with support vector regression (SVR) and its variants such as least squares SVR and LSVR. Similar generalization performance with improved or comparable learning speed to SVR and its variants demonstrates the usefulness of the proposed formulation solved by the iterative methods.

Recently, conjugate gradient methods, which usually generate descent search directions, are useful for large-scale optimization. Narushima et al. (SIAM J Optim 21:212---230, 2011) have proposed a three-term conjugate gradient method which satisfies a sufficient descent condition. We extend this method to two parameters family of three-term conjugate gradient methods which can be used to control the magnitude of the directional derivative. We show that these methods converge globally and work well for suitable choices of the parameters. Numerical results are also presented.

In this paper, a self-adjust conjugate gradient method is proposed for solving unconstrained problems, which can generate sufficient descent directions at each iteration. Different from the existent methods, a dynamical adjustment of conjugacy condition in our proposed method is developed, which can be regarded as the inheritance and development of properties of standard Hestenes---Stiefel method. Under mild condition, we show the proposed method convergent globally even if the objective function is nonconvex. Numerical results illustrate that our method can efficiently solve the test problems and therefore is promising.

In this paper, a modified Liu–Storey conjugate gradient method is proposed. The method can generate sufficient descent directions for non-linear unconstrained optimization problems. A global convergence result is established when the line search fulfils the strong Wolfe conditions. Moreover, the -linear convergence rate of the methods is proved. The extensive numerical results show that the proposed method is efficient for the unconstrained optimization problems in the CUTEr library.

Spectral modified Polak–Ribiére–Polyak projection conjugate gradient method for solving monotone systems of nonlinear equations

This article proposes a distributed optimization method for minimizing the sum of smooth and strongly convex functions with a finite communication bandwidth. Each agent has a state and an auxiliary variable to estimate the optimal solution and the average gradient of the global cost function. To cooperatively estimate the optimal solution, agents exchange the states and the auxiliary variables with their neighbors over weight-balanced networks by a dynamic encoding and decoding scheme. After the information exchanges, each agent locally updates the own state and auxiliary variable by a quantized gradient-tracking algorithm. We show that the state updated by the proposed quantized algorithm converges to the optimal solution at a linear convergence rate. We also show a sufficient condition for guaranteeing a finite communication bandwidth.