An adaptive conjugate gradient algorithm for large-scale unconstrained optimization

ABSTRACTThis paper proposes a nonmonotone scaled conjugate gradient algorithm for solving large-scale unconstrained optimization problems, which combines the idea of scaled memoryless Broyden–Fletcher–Goldfarb–Shanno preconditioned conjugate gradient method with the nonmonotone technique. An attractive property of the proposed method is that the search direction always provides sufficient descent step at each iteration. This property is independent of the line search used. Under appropriate assumptions, the method is proven to possess global convergence for nonconvex smooth functions, and R-linear convergence for strongly convex functions. Preliminary numerical results and related comparisons show the efficiency of the proposed method in practical computation.

Several questions concerning the performance in ADPCM systems of sequentially adaptive backward predictors based on the adaptive gradient and Kalman-type algorithms are addressed. Using a Jayant-type adaptive quantizer, it is shown that for bit rates less than 16 kbits/s with second order predictors and for bit rates less 18.4 kbits/s with fourth order predictors, backward-adaptive predictors have a definite performance advantage over fixed-tap predictors, since the latter may cause system divergence. For higher bit rates, the adaptive gradient predictor offers no advantage over a second order fixed-tap predictor; however, the Kalman predictor produces a substantial performance increment over the fixed-tap predictor. It is also shown that the Kalman predictor maintains a significant advantage over the adaptive gradient predictor for all bit rates from 12.8 to 32 kbits/s. Finally, it is noted that the ADPCM system divergence that occurs for fixed, multiple-tap predictors and a Jayant quantizer is caused by predictor mismatch with the input signal coupled with the infinite quantizer memory. This problem can be corrected by a modification to the quantizer adaptation logic.

This paper investigates an adaptive gradient-based distributed online convex optimization problem, wherein a group of nodes collectively optimize the global time-varying objective functions on a connected undirected graph. For such a problem, a weight decay distributed adaptive online gradient algorithm (for short, WDDAOG), by utilizing moment estimate methods and weight decay strategies, is firstly proposed. This algorithm also illustrates the difference between the weight decay and L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> regularization for distributed adaptive gradient methods. Then the dynamic regret bound for our algorithm is further analyzed. It is shown that the dynamic regret of the algorithm grows with order of O(n(1 + log T) + √nT). Numerical experiments on MNIST dataset for a multinomial logistic regression task show that WDDAOG works better than existing distributed optimization schemes.

In this paper, a self-adjust conjugate gradient method is proposed for solving unconstrained problems, which can generate sufficient descent directions at each iteration. Different from the existent methods, a dynamical adjustment of conjugacy condition in our proposed method is developed, which can be regarded as the inheritance and development of properties of standard Hestenes---Stiefel method. Under mild condition, we show the proposed method convergent globally even if the objective function is nonconvex. Numerical results illustrate that our method can efficiently solve the test problems and therefore is promising.

A Liu-Storey-type conjugate gradient method for unconstrained minimization problem with application in motion control

The adaptive s-step CG algorithm is a solver for sparse symmetric positive definite linear systems designed to reduce the synchronization cost per iteration while still achieving a user-specified accuracy requirement. In this work, we improve the adaptive s-step conjugate gradient algorithm by the use of iteratively updated estimates of the largest and smallest Ritz values, which give approximations of the largest and smallest eigenvalues of A, using a technique due to G.Meurant and P.Tichý (2018). The Ritz value estimates are used to dynamically update parameters for constructing Newton or Chebyshev polynomials so that the conditioning of the s-step bases can be continuously improved throughout the iterations. These estimates are also used to automatically set a variable related to the ratio of the sizes of the error and residual, which was previously treated as an input parameter. We show through numerical experiments that in many cases the new algorithm improves upon the previous adaptive s-step approach both in terms of numerical behavior and reduction in number of synchronizations.

In this paper, we present two families of modified three-term conjugate gradient methods for solving unconstrained large-scale smooth optimization problems. We show that our new families satisfy the Dai-Liao conjugacy condition and the sufficient descent condition under any line search technique which guarantees the positiveness of ${y_{k}^{T}} s_{k}$. For uniformly convex functions, we indicate that our families are globally convergent under weak-Wolfe-Powell line search technique and standard conditions on the objective function. We also establish a weaker global convergence theorem for general smooth functions under similar assumptions. Our numerical experiments for 260 standard problems and seven other recently developed conjugate gradient methods illustrate that the members of our families are numerically efficient and effective.

We propose a new truncated Newton method for large scale unconstrained optimization, where a Conjugate Gradient (CG)-based technique is adopted to solve Newton’s equation. In the current iteration, the Krylov method computes a pair of search directions: the first approximates the Newton step of the quadratic convex model, while the second is a suitable negative curvature direction. A test based on the quadratic model of the objective function is used to select the most promising between the two search directions. Both the latter selection rule and the CG stopping criterion for approximately solving Newton’s equation, strongly rely on conjugacy conditions. An appropriate linesearch technique is adopted for each search direction: a nonmonotone stabilization is used with the approximate Newton step, while an Armijo type linesearch is used for the negative curvature direction. The proposed algorithm is both globally and superlinearly convergent to stationary points satisfying second order necessary conditions. We carry out a significant numerical experience in order to test our proposal.

A modified Polak–Ribière–Polyak conjugate gradient algorithm for nonsmooth convex programs

An expansion of the results presented in [1], which is a comparison of least squares adaptive lattice (LSAL) and gradient adaptive lattice (GAL) linear prediction algorithms is given. New results obtained from the simulation configuration earlier described show close conformity of estimated mean reflection coefficient and squared prediction error sequences for the three algorithms. Additionally, some simulation results obtained using a single impulse superimposed on colored stationary noise and also speech-like data as lattice input processes are presented. In the former case, divergent performance of two coefficient GAL and LSAL is noted and explained.

Structural damage from earthquakes has been assessed using a variety of methodologies, both statistical and, more recently, utilizing Machine Learning (ML) algorithms. The effectiveness of data-driven procedures, even when applied to extremely time-consuming scenarios and data sets that reflect substantial expertise and research, completely depends on the quality of the underlying data. The performance of the intelligent model can also be impacted by a lack of in-depth knowledge and expertise in using complex machine learning architectures. This can also prevent some crucial hyperparameters from being adjusted, which ultimately reduces the algorithm’s reliability and generalizability. The present research offers a Bayesian-based semi-supervised Automatic Differentiation Variational Inference (ADVI) deep autoencoder for forecasting seismic damage of R/C buildings. It is a state-of-the-art, intelligent technology that automatically converts the variables in the issue into actual coordinate space using an upgraded ADVI technique. Finally, using a brand-new Adaptive Learning Rate Gradient Algorithm (ALRGA), it chooses a technique in this area that is a function of the changed variables and optimizes its parameters. Using the sophisticated ADVI technique to establish a posterior distribution without having an analytical solution is an upgraded version of the semi-supervised learning method. Estimating seismic damage to buildings is accelerated and greatly simplified by the suggested methodology, which eliminates the computational complexity of the analytical methods. By performing Nonlinear Time History Analyses of 3D R/C structures exposed to 65 earthquakes, a realistic dataset for the model evaluation is produced. The system’s strong generalizability and the proposed methodology’s detailed convergence stability reveal that it is a valuable method that can outperform other ML algorithms.

We present a broadband adaptive control method for IMT-2000 band multi-carrier power amplifiers adopting feedforward linearization. We have analyzed and implemented an error cancellation detection method employing a frequency hopping pilot and IF synchronous sampling correlator with DSP controller. An adaptive delta-modulated power gradient algorithm is used to adjust the signal and error cancellation loop control parameters. A 2.15 GHz feedforward power amplifier with digital controller is implemented. Band test results show that it covers over a 90 MHz band with more than 50 dBc of IMD at 5 MHz offset frequency for an 8.3 MHz WCDMA signal. The adaptation result shows very fast convergence.

When the flight with an optical window is flying in a high speed, the temperature and the refractive index of the air around it will be changed because of the friction, and this will make the image on the detector blurring and shifting, which will degrade the detecting precision greatly. This paper chooses a typical kind of model for the research. Ansys is used in order to gain the temperature field around the hood in real size instead of an assumptive and approximate model. So the refractive index of the air could be worked out. On this basis, the offset and the blur of the image could be figured out using the ray tracing method. Considering of the characters of the grid points distribution and the huge amount of data exported from Ansys, self-adaptive step length tracing method and interpolation method are used in order to shorten the run time and guarantee computational precision at the same time instead of the adaptive variable step method and complicated gradient algorithm. In this study, 3D simulation and programming results under different flight conditions are discussed. The results clearly show the relationship between the image quality and the flight altitude, wavelength and focal length.

In this paper, a three-dimensional subspace conjugate gradient method is proposed, in which the search direction is generated by minimizing the approximation model of the objective function in a three-dimensional subspace. The approximation model is not unique and is alternative between quadratic model and conic model by the specific criterions. The strategy of initial stepsize and nonmonotone line search are adopted, and the global convergence of the presented algorithm is established under mild assumptions. In numerical experiments, we use a collection of 80 unconstrained optimization test problems to show the competitive performance of the presented method.

The present work proposes a multi-objective improved teaching-learning based optimization (MO-ITLBO) algorithm for unconstrained and constrained multi-objective function optimization. The MO-ITLBO algorithm is the improved version of basic teaching-learning based optimization (TLBO) algorithm adapted for multi-objective problems. The basic TLBO algorithm is improved to enhance its exploration and exploitation capacities by introducing the concept of number of teachers, adaptive teaching factor, tutorial training and self-motivated learning. The MO-ITLBO algorithm uses a grid-based approach to adaptively assess the non-dominated solutions (i.e. Pareto front) maintained in an external archive. The performance of the MO-ITLBO algorithm is assessed by implementing it on unconstrained and constrained test problems proposed for the Congress on Evolutionary Computation 2009 (CEC 2009) competition. The performance assessment is done by using the inverted generational distance (IGD) measure. The IGD measures obtained by using the MO-ITLBO algorithm are compared with the IGD measures of the other state-of-the-art algorithms available in the literature. Finally, Lexicographic ordering is used to assess the overall performance of competitive algorithms. Results have shown that the proposed MO-ITLBO algorithm has obtained the 1st rank in the optimization of unconstrained test functions and the 3rd rank in the optimization of constrained test functions.