AN ENHANCED CONJUGATE GRADIENT METHOD FOR SOLVING UNCONSTRAINED OPTIMIZATION PROBLEMS

The stationary point of optimization problems can be obtained via conjugate gradient (CG) methods without the second derivative. Many researchers have used this method to solve applications in various fields, such as neural networks and image restoration. In this study, we construct a three-term CG method that fulfills convergence analysis and a descent property. Next, in the second term, we employ a Hestenses-Stiefel CG formula with some restrictions to be positive. The third term includes a negative gradient used as a search direction multiplied by an accelerating expression. We also provide some numerical results collected using a strong Wolfe line search with different sigma values over 166 optimization functions from the CUTEr library. The result shows the proposed approach is far more efficient than alternative prevalent CG methods regarding central processing unit (CPU) time, number of iterations, number of function evaluations, and gradient evaluations. Moreover, we present some applications for the proposed three-term search direction in image restoration, and we compare the results with well-known CG methods with respect to the number of iterations, CPU time, as well as root-mean-square error (RMSE). Finally, we present three applications in regression analysis, image restoration, and electrical engineering.

Abstract: The conjugate gradient (CG) method is a one of the common approaches for solving unconstrained optimization problems, notably known for its suitability for large scale problems. Many recent studies show that this method is also useful for problems of smaller scale. One of the methods used for improving the performance of CG method is hybrid approach, where a CG method is combined with another method. In this study, the ARM CG method is combined with the SMR CG method and tested under exact line search. The resulting hybrid algorithm is globally convergent under exact line search and shown to perform well numerically in comparison to other tested CG methods.

The conjugate gradient (CG) method is one of the most popular methods to solve nonlinear unconstrained optimization problems. The Hestenes-Stiefel (HS) CG formula is considered one of the most efficient methods developed in this century. In addition, the HS coefficient is related to the conjugacy condition regardless of the line search method used. However, the HS parameter may not satisfy the global convergence properties of the CG method with the Wolfe-Powell line search if the descent condition is not satisfied. In this paper, we use the original HS CG formula with a mild condition to construct a CG method with restart using the negative gradient. The convergence and descent properties with the strong Wolfe-Powell (SWP) and weak Wolfe-Powell (WWP) line searches are established. Using this condition, we guarantee that the HS formula is non-negative, its value is restricted, and the number of restarts is not too high. Numerical computations with the SWP line search and some standard optimization problems demonstrate the robustness and efficiency of the new version of the CG parameter in comparison with the latest and classical CG formulas. An example is used to describe the benefit of using different initial points to obtain different solutions for multimodal optimization functions.

<abstract> <p>The conjugate gradient (CG) method is a method to solve unconstrained optimization problems. Moreover CG method can be applied in medical science, industry, neural network, and many others. In this paper a new three term CG method is proposed. The new CG formula is constructed based on DL and WYL CG formulas to be non-negative and inherits the properties of HS formula. The new modification satisfies the convergence properties and the sufficient descent property. The numerical results show that the new modification is more efficient than DL, WYL, and CG-Descent formulas. We use more than 200 functions from CUTEst library to compare the results between these methods in term of number of iterations, function evaluations, gradient evaluations, and CPU time.</p> </abstract>

At present, the conjugate gradient (CG) method of Hager and Zhang (Hager and Zhang, SIAM Journal on Optimization, 16(2005)) is regarded as one of the most effective CG methods for optimization problems. In order to further study the CG method, we develop the Hager and Zhang's CG method and present two modified CG formulas, where the given formulas possess the value information of not only the gradient but also the function. Moreover, the sufficient descent condition will be holden without any line search. The global convergence is established for nonconvex function under suitable conditions. Numerical results show that the proposed methods are competitive to the normal conjugate gradient method.

Background/Objectives: The Conjugate Gradient (CG) methods are the well-known iterative methods use for finding solutions to nonlinear system equations. There is need to address the jamming phenomenal facing the current class of this methods. Methods/Statistical Analysis: In order to address the shortcomings, we work on the denominator of the Yao et al., CG method which is known to generate descent direction for objective functions by proposing an entire different CG coefficient which can easily switch in case jamming occurs by imposing some parameters thereby guarantee global convergence. Findings: The proposed CG formula performs better than classical methods as well as Yao et al. Under Wolfe line search condition, the convergence analysis of the proposed CG formula was established. Some benchmark problems from cute collections are used as basis of strength comparisons of the proposed formula against some other CG formulas. Effectiveness and efficiency of the obtained results for the proposed formula is clearly shown by adopting the performance profile of Dolan and More’ which is one of most acceptable techniques of strength comparisons among methods. Application: Mathematicians and Engineers who are interested in finding solutions to large scale nonlinear equations can apply the method leading to global optimization dealing with best possible solutions ever for given problems.

PurposeThe purpose of this paper is to apply the conjugate gradient (CG) method, together with the adjoint operator (AO) to the pulsating heat pipe problem, including some quite interesting experimental results. The CG method, together with the AO, was able to estimate the unknown functions more efficiently than the other techniques presented in this paper. The estimation of local heat transfer coefficients, rather than the global ones, in pulsating heat pipes is a relatively new subject and presenting a robust, efficient and self-regularized inverse tool to estimate it, supported also by some experimental results, is the main purpose of this paper. To also increase the visibility and the general use of the paper to the heat transfer community, the authors include, as supplemental material, all numerical and experimental data used in this paper.Design/methodology/approachThe approach was established on the solution of the inverse heat conduction problem in the wall by using as starting data the temperature measurements on the outer surface. The procedure is based on the CG method with AO. The here proposed approach was first verified adopting synthetic data and then it was validated with real cases regarding pulsating heat pipes.FindingsAn original fast methodology to estimate local convective heat flux is proposed. The procedure has been validated both numerically and experimentally. The procedure has been compared to other classical methods presenting some peculiar benefits.Practical implicationsThe approach is suitable for pulsating heat pipes performance evaluation because these devices present a local heat flux distribution characterized by an important variation both in time and in space as a result of the complex flow patterns that are generated in this type of devices.Originality/valueThe procedure here proposed shows these benefits: it affords a general model of the heat conduction problem that is effortlessly customized for the particular case, it can be applied also to large datasets and it presents reduced computational expense.

A generalized data windowing scheme for adaptive conjugate gradient algorithms

We consider the bilinear optimal control of an advection-reaction-diffusion system, where the control arises as the velocity field in the advection term. Such a problem is generally challenging from both theoretical analysis and algorithmic design perspectives, mainly because the state variable depends nonlinearly on the control variable and, an additional divergence-free constraint on the control is coupled together with the state equation. Mathematically, the proof of the existence of optimal solutions is delicate, and, up to now, only some results have been known for a few special cases where additional restrictions are imposed on the space dimension and the regularity of the control. We prove the existence of optimal controls and derive the first-order optimality conditions in general settings without any extra assumptions. Computationally, the well-known conjugate gradient (CG) method can be applied conceptually. However, due to the additional divergence-free constraint on the control variable and the nonlinear relation between the state and control variables, it is challenging to compute the gradient and the optimal stepsize at each CG iteration, and thus nontrivial to implement the CG method. To address these issues, we advocate a fast inner preconditioned CG method to ensure the divergence-free constraint and an efficient inexactness strategy to determine an appropriate stepsize. An easily implementable nested CG method is thus proposed for solving such a complicated problem. For the numerical discretization, we combine finite difference methods for the time discretization and finite element methods for the space discretization. Efficiency of the proposed nested CG method is promisingly validated by the results of some preliminary numerical experiments.

Although numerical calculations of heat transfer and fluid flow can provide detailed insights into welding processes and welded materials, these calculations are complex and unsuitable in situations where rapid calculations are needed. A recourse is to train and validate a neural network, using results from a well tested heat and fluid flow model to significantly expedite calculations and ensure that the computed results conform to the basic laws of conservation of mass, momentum and energy. Seven feedforward neural networks were developed for gas metal arc (GMA) fillet welding, one each for predicting penetration, leg length, throat, weld pool length, cooling time between 800°C and 500°C, maximum velocity and peak temperature in the weld pool. Each model considered 22 inputs that included all the welding variables, such as current, voltage, welding speed, wire radius, wire feed rate, arc efficiency, arc radius, power distribution, and material properties such as thermal conductivity, specific heat and temperature coefficient of surface tension. The weights in the neural network models were calculated using the conjugate gradient (CG) method and by a hybrid optimisation scheme involving the CG method and a genetic algorithm (GA). The neural network produced by the hybrid optimisation model produced better results than the networks based on the CG method with various sets of randomised initial weights. The CG method alone was unable to find the best optimal weights for achieving low errors. The hybrid optimisation scheme helped in finding optimal weights through a global search, as evidenced by good agreement between all the outputs from the neural networks and the corresponding results from the heat and fluid flow model.

This study introduces an improved three-term conjugate gradient (CG) algorithm which fulfills the descent conditions and exhibits good global convergence properties. The development of this new algorithm is based on the findings of recently introduced generalized RMIL CG technique. The algorithm has been modified so that the search direction would always meet the sufficient descent condition no matter the line-search techniques employed. Under some given assumptions, the algorithm’s global convergence result for the general nonconvex functions is established. The numerical efficacy of the suggested three-term CG algorithm is assessed through comparisons with four other CG algorithms. The assessment is performed using an array of unconstrained optimization benchmark functions. The acquired findings indicate that all the considered algorithms exhibit similar performance, particularly the variants of the RMIL method. One algorithm is more robust and slightly faster than the rest. However, it is essential to highlight that the newly developed CG formula outperforms all the other algorithms, including a recently presented three-term PRP algorithm. The study further expanded the proposed method to address an image restoration problem. The computational experiments yielded promising results, proving that the proposed CG algorithm surpasses others. It produces higher-quality output images, requires less CPU time (sec), and achieves higher PSNR values.

The standard implementation of the conjugate gradient algorithm suffers from communication bottlenecks on parallel architectures, due primarily to the two global reductions required every iteration. In this paper, we study conjugate gradient variants which decrease the runtime per iteration by overlapping global synchronizations, and in the case of pipelined variants, matrix-vector products. Through the use of a predict-and-recompute scheme, whereby recursively-updated quantities are first used as a predictor for their true values and then recomputed exactly at a later point in the iteration, these variants are observed to have convergence behavior nearly as good as the standard conjugate gradient implementation on a variety of test problems. We provide a rounding error analysis which provides insight into this observation. It is also verified experimentally that the variants studied do indeed reduce the runtime per iteration in practice and that they scale similarly to previously-studied communication-hiding variants. Finally, because these variants achieve good convergence without the use of any additional input parameters, they have the potential to be used in place of the standard conjugate gradient implementation in a range of applications.

The left-preconditioned communication avoiding conjugate gradient (LP-CA-CG) method is applied to the pressure Poisson equation in the multiphase CFD code JUPITER. The arithmetic intensity of the LP-CA-CG method is analyzed, and is dramatically improved by loop splitting for inner product operations and for three term recurrence operations. Two LPCA-CG solvers with block Jacobi preconditioning and with underlap preconditioning are developed. The former is developed based on a hybrid CA approach, in which CA is applied only to global collective communications for inner product operations. The latter is a full CA approach, in which CA is applied also to local point-to-point communications in sparse matrix-vector (SpMV) operations and preconditioning. CA-SpMV requires additional computation for overlapping regions. CA-preconditiong is enabled by underlap preconditioning, which approximates preconditioning for overlapping regions by point Jacobi preconditioning. It is shown that on the K computer, the former is faster, because the performance of local point-to-point communications scales well, and the convergence property becomes worse with underlap preconditioning. The LP-CA-CG solver shows good strong scaling up to 30,000 nodes, where the LP-CA-CG solver achieved higher performance than the original CG solver by reducing the cost of global collective communications by 69 percent.

Self-scaled conjugate gradient training algorithms

Presently, a preferred minimization for strong‐constraint four‐dimensional variational (4D‐Var) assimilation uses a Lanczos‐based conjugate gradient (CG) algorithm. This requires the availability of a square‐root of the background‐error covariance matrix (B). In the context of weak‐constraint 4D‐Var, this requirement might be too restrictive for the formulations of the model error term. It might therefore be desirable to avoid a square‐root decomposition of the augmented background term. An appealing minimization scheme is the double CG minimization employed, for example, in the grid‐point statistical interpolation (GSI) analysis. Realizing the double CG algorithm is a special case of the more general bi‐conjugate gradient (BiCG) method for solving non‐symmetric problems, the present work introduces a Lanczos‐based preconditioning strategy when B, instead of its square‐root, is used initially. Implementation of the scheme is done in the context of the GSI analysis system, and preliminary experiments are presented using its 3D‐Var version. Comparison of the Lanczos‐based CG and the BiCG shows that the algorithms converge at the same rate and to the same solution. Despite the additional computational cost, the importance of the re‐orthogonalization step is also shown to be fundamental to any of these CG algorithms. Furthermore, when using the Hessian eigenvectors for preconditioning, the BiCG behaviour is shown to be comparable to that of the Lanczos‐CG algorithm. Both schemes construct the same approximation of the Hessian with the same number of eigenvectors, and benefit in the same way from the reduction of the condition number. The efficiency, computational cost, and stability of the three algorithms are discussed. Copyright © 2012 Royal Meteorological Society