An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations

A stochastic conjugate gradient multi-modulus algorithm based on global artificial fish swarm and genetic optimization DNA encoding sequences(GAFS-GDNA-SMMA) is proposed to solve the high computational loads, low convergence rate, and large mean square error(MSE) of the multi-modulus blind equalization algorithm(MMA). In this proposed algorithm, the optimal DNA coding sequence can be found and used as the initial optimal weight vector of the stochastic conjugate gradient MMA(SMMA) after decoding the optimization DNA sequences. The simulation results show that the proposed algorithm has the faster convergence speed and smaller mean square error comparison with MMA, SMMA, GA-DNA-SMMA(Genetic algorithm and DNA optimization based SMMA)

A matrix-free method for constrained equations is proposed, which is a combination of the well-known PRP (Polak-Ribière-Polyak) conjugate gradient method and the famous hyperplane projection method. The new method is not only derivative-free, but also completely matrix-free, and consequently, it can be applied to solve large-scale constrained equations. We obtain global convergence of the new method without any differentiability requirement on the constrained equations. Compared with the existing gradient methods for solving such problem, the new method possesses linear convergence rate under standard conditions, and a relax factorγis attached in the update step to accelerate convergence. Preliminary numerical results show that it is promising in practice.

At present, the conjugate gradient (CG) method of Hager and Zhang (Hager and Zhang, SIAM Journal on Optimization, 16(2005)) is regarded as one of the most effective CG methods for optimization problems. In order to further study the CG method, we develop the Hager and Zhang's CG method and present two modified CG formulas, where the given formulas possess the value information of not only the gradient but also the function. Moreover, the sufficient descent condition will be holden without any line search. The global convergence is established for nonconvex function under suitable conditions. Numerical results show that the proposed methods are competitive to the normal conjugate gradient method.

Background:
 A new optimization algorithm is presented. The method is biased with new non monotone line search an accelerated three term conjugate gradient method damped of Quasi Newton method compared to previews method design efficiency in provident of more than one factor for different optimization problem are more dramatic due to the ability of the technique to utile existing data.
 Materials and Methods:
 New monotone line search, new monotone line search, new modification of Damped Quasi-Newton method, Motivation and New Quasi-Newton Algorithm (MQ) and Global convergence.
 Results:
 In this work, we have n tendency to compare our new algorithm with same classical strategies like [7] by exploiting of unconstrained nonlinear optimization problem the functions obtained from Andrei [5, 6] Waziri and Sabiu (2015)[10] and La couzetul (2004)[3]. The numerical experiments demonstrate the performance of the proposed method. We selected seven relatively unconstrained problems with the size varies from 10 to 100. We consider the three sizes of each problem so that the total number of problem is 21 test problems. We stop the iteration when is satisfied All codes were written in Matlab R2017a and run on a pc with Intel COREi4 with a processor with 4GB of Ram and CPU 2.3GHZ we solved test problems using two different initial starting points.
 Conclusion:
 In this research article, a project on an accelerated three-term efficient algorithm for numerical optimization has presented the method as completely a derivative-free algorithm with less NOI and NOF and CPU time computed to the existing methods .using classical assumption the global convergence was also proved. Numerical results using the three terms efficient algorithm show that the algorithm is promising.

We present a new solution framework to solve the generalized trust region subproblem (GTRS) of minimizing a quadratic objective over a quadratic constraint. More specifically, we derive a convex quadratic reformulation (CQR) via minimizing a linear objective over two convex quadratic constraints for the GTRS. We show that an optimal solution of the GTRS can be recovered from an optimal solution of the CQR. We further prove that this CQR is equivalent to minimizing the maximum of the two convex quadratic functions derived from the CQR for the case under investigation. Although the latter minimax problem is nonsmooth, it is well structured and convex. We thus develop two steepest descent algorithms corresponding to two different line search rules. We prove global sublinear convergence rates for both algorithms. We also obtain a local linear convergence rate of the first algorithm by estimating the Kurdyka-Łojasiewicz exponent at any optimal solution under mild conditions. We finally demonstrate the efficiency of our algorithms with numerical experiments.

In this paper, we introduce a three-term conjugate gradient-type projection method for solving constrained monotone nonlinear equations. In this method, we firstly undertake the transformation of the relative matrices proposed by Yao and Ning. Secondly, we obtain the new relative matrices involving two parameters. Subsequently, we construct a relationship for the two parameters via the quasi-Newton equation and obtain the parameters by simplifying maximum eigenvalue of the new relative matrices. Finally, combining the three-term conjugate gradient method with projection technique, we establish an efficient three-term conjugate gradient-type projection algorithm. Meanwhile, we also give some theoretical analysis about the global convergence and R-linear convergence of the proposed algorithm under quite reasonable technical assumptions. Performance comparisons show that our proposed method is competitive and efficient for solving large-scale nonlinear monotone equations with convex constraints. Furthermore, the presented algorithm is also applied to recovery of a sparse signal in compressive sensing, and obtain practical, efficient and competitive performance in comparing with state-of-the-art algorithms.

Self-scaled conjugate gradient training algorithms

The global convergence of the Polak–Ribière–Polyak conjugate gradient algorithm under inexact line search for nonconvex functions

In this article, we try to proposed a new conjugate gradient method for solving unconstrained optimization problems, we focus on conjugate gradient methods applied to the non-linear unconstrained optimization problems, the positive step size is obtained by a line search and the new scalar to the new direction for the conjugate gradient method is derived from the quadratic function and Taylor series and by using quasi newton condition and Newton direction while deriving the new formulae. We also prove that the search direction of the new conjugate gradient method satisfies the sufficient descent and all assumptions of the global convergence property are considered and proved .in order to complete the benefit of our research we should take into account studied the numerical results which are written in FORTRAN language when the objective function is compared our new algorithm with HS and PRP methods on the similar set of unconstrained optimization test problems which is very efficient and encouragement numerical results.

Non-convex optimization problems are prevalent in scientific and engineering applications, often characterized by multiple local minima that challenge standard optimization techniques. Global optimization methods, such as Genetic Algorithms (GA) (1992) [4] and Particle Swarm Optimization (PSO) (1995) [6], provide robust global exploration but tend to converge slowly. Conversely, local search methods, such as Newton’s method (2004) [1], (2006) [7], and the Conjugate Gradient (CG) (1964) [3] method, offer rapid convergence but are susceptible to local minima. In this paper, we present four hybrid optimization algorithms that integrate the exploratory strength of global algorithms with the refinement ability of local methods. The proposed hybrids, Genetic Algorithms (GA) + Newton, GA + CG, PSO + Newton, and PSO + CG, are evaluated against their standalone counterparts on standard benchmark functions (2013) [5], including Rosenbrock, Rastrigin, Ackley, and Himmelblau. This work builds upon our previous research on the hybridization of two global search algorithms, the Nelder–Mead (simplex) algorithm and the Bat algorithm, which resulted in the Hybrid Simplex Bat Algorithm (HSBA) [2], recently Published. Results demonstrate that the hybrid approaches consistently outperform traditional methods in terms of accuracy, convergence rate, and computational efficiency.

An adaptive array code acquisition for direct-sequence/ code-division multiple access (DS/CDMA) systems was recently proposed to enhance the performance of the conventional correlator-based method. The scheme consists of an adaptive spatial and an adaptive temporal filter, and can simultaneously perform beamforming and code-delay estimation. Unfortunately, the scheme uses a least-mean-square (LMS) adaptive algorithm, and its convergence is slow. Although the recursive-least-squares (RLS) algorithm can be applied, the computational complexity will greatly increase. In this paper, we solve the dilemma with a low-complexity conjugate gradient (LCG) algorithm, which can be considered as a special case of a modified conjugate gradient (MCG) algorithm. Unlike the original conjugate gradient (CG) algorithm developed for adaptive applications, the proposed method, exploiting the special structure inherent in the input correlation matrix, requires a low computational-complexity. It can be shown that the computational complexity of the proposed method is on the same order of the LMS algorithm. However, the convergence rate is improved significantly. Simulation results show that the performance of adaptive array code acquisition with the proposed CG algorithm is comparable to that with the original CG algorithm.

Abstract: The conjugate gradient (CG) method is a one of the common approaches for solving unconstrained optimization problems, notably known for its suitability for large scale problems. Many recent studies show that this method is also useful for problems of smaller scale. One of the methods used for improving the performance of CG method is hybrid approach, where a CG method is combined with another method. In this study, the ARM CG method is combined with the SMR CG method and tested under exact line search. The resulting hybrid algorithm is globally convergent under exact line search and shown to perform well numerically in comparison to other tested CG methods.

This study introduces an improved three-term conjugate gradient (CG) algorithm which fulfills the descent conditions and exhibits good global convergence properties. The development of this new algorithm is based on the findings of recently introduced generalized RMIL CG technique. The algorithm has been modified so that the search direction would always meet the sufficient descent condition no matter the line-search techniques employed. Under some given assumptions, the algorithm’s global convergence result for the general nonconvex functions is established. The numerical efficacy of the suggested three-term CG algorithm is assessed through comparisons with four other CG algorithms. The assessment is performed using an array of unconstrained optimization benchmark functions. The acquired findings indicate that all the considered algorithms exhibit similar performance, particularly the variants of the RMIL method. One algorithm is more robust and slightly faster than the rest. However, it is essential to highlight that the newly developed CG formula outperforms all the other algorithms, including a recently presented three-term PRP algorithm. The study further expanded the proposed method to address an image restoration problem. The computational experiments yielded promising results, proving that the proposed CG algorithm surpasses others. It produces higher-quality output images, requires less CPU time (sec), and achieves higher PSNR values.

Solving optimization problems with sparse or low-rank optimal solutions has been an important topic since the recent emergence of compressed sensing and its matrix extensions such as the matrix rank minimization and robust principal component analysis problems. Compressed sensing enables one to recover a signal or image with fewer observations than the “length” of the signal or image, and thus provides potential breakthroughs in applications where data acquisition is costly. However, the potential impact of compressed sensing cannot be realized without efficient optimization algorithms that can handle extremely large-scale and dense data from real applications. Although the convex relaxations of these problems can be reformulated as either linear programming, second-order cone programming or semidefinite programming problems, the standard methods for solving these relaxations are not applicable because the problems are usually of huge size and contain dense data. In this dissertation, we give efficient algorithms for solving these “sparse” optimization problems and analyze the convergence and iteration complexity properties of these algorithms. Chapter 2 presents algorithms for solving the linearly constrained matrix rank minimization problem. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast and solved as a semidefinite programming problem, such an approach is computationally expensive when the matrices are large. In Chapter 2, we propose fixed-point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems. Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 minutes by sampling only 20 percent of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms. In Chapter 3, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving linearly constrained matrix rank minimization problems are reported. Chapters 4 and 5 considers alternating direction type methods for solving composite convex optimization problems. We present in Chapter 4 alternating linearization algorithms that are based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions. Our basic methods require at most O(1/e) iterations to obtain an e-optimal solution, while our accelerated (i.e., fast) versions require at most O(1/ 3 ) iterations, with little change in the computational effort required at each iteration. For more general problem, i.e., minimizing the sum of K convex functions, we propose multiple-splitting algorithms for solving them. We propose both basic and accelerated algorithms with O(1/e) and O(1/ 3 ) iteration complexity bounds for obtaining an e-optimal solution. To the best of our knowledge, the complexity results presented in these two chapters are the first ones of this type that have been given for splitting and alternating direction type methods. Numerical results on various applications in sparse and low-rank optimization, including compressed sensing, matrix completion, image deblurring, robust principal component analysis, are reported to demonstrate the efficiency of our methods.

Although numerical calculations of heat transfer and fluid flow can provide detailed insights into welding processes and welded materials, these calculations are complex and unsuitable in situations where rapid calculations are needed. A recourse is to train and validate a neural network, using results from a well tested heat and fluid flow model to significantly expedite calculations and ensure that the computed results conform to the basic laws of conservation of mass, momentum and energy. Seven feedforward neural networks were developed for gas metal arc (GMA) fillet welding, one each for predicting penetration, leg length, throat, weld pool length, cooling time between 800°C and 500°C, maximum velocity and peak temperature in the weld pool. Each model considered 22 inputs that included all the welding variables, such as current, voltage, welding speed, wire radius, wire feed rate, arc efficiency, arc radius, power distribution, and material properties such as thermal conductivity, specific heat and temperature coefficient of surface tension. The weights in the neural network models were calculated using the conjugate gradient (CG) method and by a hybrid optimisation scheme involving the CG method and a genetic algorithm (GA). The neural network produced by the hybrid optimisation model produced better results than the networks based on the CG method with various sets of randomised initial weights. The CG method alone was unable to find the best optimal weights for achieving low errors. The hybrid optimisation scheme helped in finding optimal weights through a global search, as evidenced by good agreement between all the outputs from the neural networks and the corresponding results from the heat and fluid flow model.