A nonmonotone truncated Newton–Krylov method exploiting negative curvature directions, for large scale unconstrained optimization

In this paper we deal with the iterative computation of negative curvature directions of an objective function, within large scale optimization frameworks. In particular, suitable directions of negative curvature of the objective function represent an essential tool, to guarantee convergence to second order critical points. However, an "adequate" negative curvature direction is often required to have a good resemblance to an eigenvector corresponding to the smallest eigenvalue of the Hessian matrix. Thus, its computation may be a very difficult task on large scale problems. Several strategies proposed in literature compute such a direction relying on matrix factorizations, so that they may be inefficient or even impracticable in a large scale setting. On the other hand, the iterative methods proposed either need to store a large matrix, or they need to rerun the recurrence. On this guideline, in this paper we propose the use of an iterative method, based on a planar Conjugate Gradient scheme. Under mild assumptions, we provide theory for using the latter method to compute adequate negative curvature directions, within optimization frameworks. In our proposal any matrix storage is avoided, along with any additional rerun.

In this paper we propose efficient new linesearch algorithms for solving large scale unconstrained optimization problems which exploit any local nonconvexity of the objective function. Current algorithms in this class typically compute a pair of search directions at every iteration: a Newton-type direction, which ensures both global and fast asymptotic convergence, and a negative curvature direction, which enables the iterates to escape from the region of local non-convexity. A new point is generated by performing a search along a line or a curve obtained by combining these two directions. However, in almost all if these algorithms, the relative scaling of the directions is not taken into account. We propose a new algorithm which accounts for the relative scaling of the two directions. To do this, only the most promising of the two directions is selected at any given iteration, and a linesearch is performed along the chosen direction. The appropriate direction is selected by estimating the rate of decrease of the quadratic model of the objective function in both candidate directions. We prove global convergence to second-order critical points for the new algorithm, and report some preliminary numerical results.

Conjugate gradient methods are an important class of methods for solving unconstrained optimization problems, especially for large-scale problems. Recently, they have been much studied. In this paper, we further study the conjugate gradient method for unconstrained optimization. We focus our attention to the descent conjugate gradient method. This paper presents a modified conjugate gradient method. An interesting feature of the presented method is that the direction is always a descent direction for the objective function. Moreover, the property is independent of the line search used. Under mild conditions, we prove that the modified conjugate gradient method with Armijo-type line search is globally convergent. We also present some numerical results to show the efficiency of the proposed method. DOI: http://dx.doi.org/10.11591/telkomnika.v11i11.2894

This study introduces an improved three-term conjugate gradient (CG) algorithm which fulfills the descent conditions and exhibits good global convergence properties. The development of this new algorithm is based on the findings of recently introduced generalized RMIL CG technique. The algorithm has been modified so that the search direction would always meet the sufficient descent condition no matter the line-search techniques employed. Under some given assumptions, the algorithm’s global convergence result for the general nonconvex functions is established. The numerical efficacy of the suggested three-term CG algorithm is assessed through comparisons with four other CG algorithms. The assessment is performed using an array of unconstrained optimization benchmark functions. The acquired findings indicate that all the considered algorithms exhibit similar performance, particularly the variants of the RMIL method. One algorithm is more robust and slightly faster than the rest. However, it is essential to highlight that the newly developed CG formula outperforms all the other algorithms, including a recently presented three-term PRP algorithm. The study further expanded the proposed method to address an image restoration problem. The computational experiments yielded promising results, proving that the proposed CG algorithm surpasses others. It produces higher-quality output images, requires less CPU time (sec), and achieves higher PSNR values.

The Spectral conjugate gradient (SCG) methods are among the efficient variants of CG algorithms which are obtained by combining the spectral gradient parameter and CG parameter. The success of SCG methods relies on effective choices of the step-size $\alpha _{k}$ and the search direction $d_{k}$ . This paper presents an SCG method for unconstrained optimization models. The search directions generated by the new method possess sufficient descent property without the restart condition and independent of the line search procedure used. The global convergence of the new method is proved under the weak Wolfe line search. Preliminary numerical results are presented which show that the method is efficient and promising, particularly for large-scale problems. Also, the method was applied to solve the robotic motion control problem and portfolio selection problem.

An adaptive conjugate gradient algorithm for large-scale unconstrained optimization

<abstract><p>The conjugate gradient (CG) method is an optimization method, which, in its application, has a fast convergence. Until now, many CG methods have been developed to improve computational performance and have been applied to real-world problems. In this paper, a new hybrid three-term CG method is proposed for solving unconstrained optimization problems. The search direction is a three-term hybrid form of the Hestenes-Stiefel (HS) and the Polak-Ribiére-Polyak (PRP) CG coefficients, and it satisfies the sufficient descent condition. In addition, the global convergence properties of the proposed method will also be proved under the weak Wolfe line search. By using several test functions, numerical results show that the proposed method is most efficient compared to some of the existing methods. In addition, the proposed method is used in practical application problems for image restoration and portfolio selection.</p></abstract>

The stationary point of optimization problems can be obtained via conjugate gradient (CG) methods without the second derivative. Many researchers have used this method to solve applications in various fields, such as neural networks and image restoration. In this study, we construct a three-term CG method that fulfills convergence analysis and a descent property. Next, in the second term, we employ a Hestenses-Stiefel CG formula with some restrictions to be positive. The third term includes a negative gradient used as a search direction multiplied by an accelerating expression. We also provide some numerical results collected using a strong Wolfe line search with different sigma values over 166 optimization functions from the CUTEr library. The result shows the proposed approach is far more efficient than alternative prevalent CG methods regarding central processing unit (CPU) time, number of iterations, number of function evaluations, and gradient evaluations. Moreover, we present some applications for the proposed three-term search direction in image restoration, and we compare the results with well-known CG methods with respect to the number of iterations, CPU time, as well as root-mean-square error (RMSE). Finally, we present three applications in regression analysis, image restoration, and electrical engineering.

In this paper, we present a new hybrid conjugate gradient algorithm for unconstrained optimization. This method is a convex combination of Liu-Storey conjugate gradient method and Fletcher-Reeves conjugate gradient method. We also prove that the search direction of any hybrid conjugate gradient method, which is a convex combination of two conjugate gradient methods, satisfies the famous D-L conjugacy condition and in the same time accords with the Newton direction with the suitable condition. Furthermore, this property doesn’t depend on any line search. Next, we also prove that, moduling the value of the parameter t, the Newton direction condition is equivalent to Dai-Liao conjugacy condition. The strong Wolfe line search conditions are used. The global convergence of this new method is proved. Numerical comparisons show that the present hybrid conjugate gradient algorithm is the efficient one.

Self-scaled conjugate gradient training algorithms

In this paper, we present two families of modified three-term conjugate gradient methods for solving unconstrained large-scale smooth optimization problems. We show that our new families satisfy the Dai-Liao conjugacy condition and the sufficient descent condition under any line search technique which guarantees the positiveness of ${y_{k}^{T}} s_{k}$. For uniformly convex functions, we indicate that our families are globally convergent under weak-Wolfe-Powell line search technique and standard conditions on the objective function. We also establish a weaker global convergence theorem for general smooth functions under similar assumptions. Our numerical experiments for 260 standard problems and seven other recently developed conjugate gradient methods illustrate that the members of our families are numerically efficient and effective.

In this paper, a three-dimensional subspace method is proposed, in which the search direction is generated by minimizing the approximation model of the objective function in a three-dimensional subspace. The approximation model of the objective function is not unique, and alternatives can be chosen between a symmetric quadratic model and a conic model by specific criteria. Moreover, the idea of a WLY conjugate gradient method is applied to characterize the change of gradient direction between adjacent iteration points. The strategy of initial stepsize and nonmonotone line search are adopted, and the global convergence of the presented algorithm is established under mild assumptions. In numerical experiments, we use a collection of 80 unconstrained optimization test problems to show the competitive performance of the presented method.

Full Waveform Inversion (FWI) is a seismic imaging technique that utilizes the entire content of seismic trace. This method aims to minimizes differences between real and synthetic seismograms. FWI consists of three main steps: forward modelling to simulate synthetic seismograms, adjoint simulation to calculate the gradient misfit, and optimization to obtain an appropriate model perturbation. FWI is an extremely non-linear inversion technique and computationally intensive since it involves many model parameters. In the context of geophysics, finding parameter models that describe the structure of the earth’s subsurface is the main goal. Geophysical measurements in the field only obtain data that is influenced by the parameter model, in this case seismic recording data, therefore mathematical techniques are studied and applied to determine the correct parameter model from the data obtained, also known as the inversion process. In this study, main research area is on optimization step which include mathematical approach to find perturbation model effectively. The Newton method can be used in the optimization step of FWI. However, its implementation is not promising due to high computational cost in calculating Hessian matrix. To reduce computation cost, an identity matrix is chosen to replace the Hessian matrix, as it usually done in the steepest descent method. Since FWI has a high level of non-linearity, steepest descent method requires many iterations to minimize the objective function. Therefore, this study uses a conjugate gradient method to make the data misfit minimized more quickly. Apart from that, the Line search technique is used to find an appropriate step-length at each iteration on both methods. The FWI workflow is applied to the synthetic acoustic velocity model. We conduct FWI using steepest descent and conjugate gradient methods. The results show that the conjugate gradient method requires fewer iterations than the steepest descent method to minimize the objective function. FWI model result using conjugate gradient method is closer with the true model compared to steepest descent method, showed on misfit result and model image. By comparing these methods we can see that conjugate gradient method can be chosen for application on FWI optimization step for better result on seismic imaging. On the industry scale, better results provide less uncertainty on the model parameters that useful for energy exploration or engineering.

This paper addresses the question of whether it can be beneficial for an optimization algorithm to follow directions of negative curvature. Although prior work has established convergence results for algorithms that integrate both descent and negative curvature steps, there has not yet been extensive numerical evidence showing that such methods offer consistent performance improvements. In this paper, we present new frameworks for combining descent and negative curvature directions: alternating two-step approaches and dynamic step approaches. The aspect that distinguishes our approaches from ones previously proposed is that they make algorithmic decisions based on (estimated) upper-bounding models of the objective function. A consequence of this aspect is that our frameworks can, in theory, employ fixed stepsizes, which makes the methods readily translatable from deterministic to stochastic settings. For deterministic problems, we show that instances of our dynamic framework yield gains in performance compared to related methods that only follow descent steps. We also show that gains can be made in a stochastic setting in cases when a standard stochastic-gradient-type method might make slow progress.

In this thesis we study the problem of minimizing nonlinear functions of several variables, where the objective function is continuously differentiable on an open subset of Rn. We develop mathematical optimization methods for solving large scale problems, i.e., problems whose variables are many thousands, even millions. The proposed method is based on the theoretical study of the properties of minimal and low memory Quasi-Newton updates. We establish theorems concerning the characteristic polynomial, the number of distinct eigenvalues and corresponding eigenvectors. We derive closed formulas for calculating these quantities, avoiding both the storage and factorization of matrices. The new theoretical results are applied in the large scale trust region subproblem for calculating nearly exact solutions as well as in a curvilinear search that uses a Quasi-Newton and a negative curvature direction. The new method is drastically reducing the spatial complexity of known algorithms of nonlinear programming. As a result, the new algorithms have spatial complexity Θ(n), while they are maintaining good convergence properties. The numerical results show that the proposed algorithms are efficient, fast and very effective when used in solving large scale problems.