Shape-changing trust-region methods using multipoint symmetric secant matrices

In large-scale optimization, when either forming or storing Hessian matrices are prohibitively expensive, quasi-Newton methods are often used in lieu of Newton's method because they only require first-order information to approximate the true Hessian. Multipoint symmetric secant (MSS) methods can be thought of as generalizations of quasi-Newton methods in that they attempt to impose additional requirements on their approximation of the Hessian. Given an initial Hessian approximation, MSS methods generate a sequence of possibly-indefinite matrices using rank-2 updates to solve nonconvex unconstrained optimization problems. For practical reasons, up to now, the initialization has been a constant multiple of the identity matrix. In this paper, we propose a new limited-memory MSS method for large-scale nonconvex optimization that allows for dense initializations. Numerical results on the CUTEst test problems suggest that the MSS method using a dense initialization outperforms the standard initialization. Numerical results also suggest that this approach is competitive with both a basic L-SR1 trust-region method and an L-PSB method.

It is well known that among the current methods for unconstrained optimization problems the quasi-Newton methods with global strategy may be the most efficient methods, which have local superlinear convergence. However, when the iterative point is far away from the solution of the problem, quasi-Newton method may proceed slowly for the general unconstrained optimization problems. In this article an adaptive conic trust-region method for unconstrained optimization is presented. Not only the gradient information but also the values of the objective function are used to construct the local model at the current iterative point. Moreover, we define a concept of super steepest descent direction and embed its information into the local model. The amount of computation in each iteration of this adaptive algorithm is the same as that of the standard quasi-Newton method with trust region. Some numerical results show that the modified method requires fewer iterations than the standard methods to reach the solution of the optimization problem. Global and local convergence of the method is also analyzed.

Trust-region based solver for nonlinear transport in heterogeneous porous media

A variant of trust-region methods for unconstrained optimization

This paper presents a hybrid spectral conjugate gradient method for large-scale unconstrained optimization, which possesses a self-adjusting property. Under the standard Wolfe conditions, its global convergence result is established. Preliminary numerical results are reported on a set of large-scale problems in CUTEr to show the convergence and efficiency of the proposed method.

A counter-example is given to several recently published results on duality bound methods for nonconvex global optimization.

This research falls into the gap between applied statistics and numerical optimization in a specific topic—Ridge Analysis (RA). This article proposes using the trust-region (TR) methods in numerical optimization to solve the RA problem, arising from the literature of response surface methodology (RSM) in applied statistics, where its goal is to help engineers for ‘process improvement’ to find the better response value of the predicted response function within the boundary of experimentation. In the field of numerical optimization, as the family of TR approaches always exhibits excellent mathematical properties during optimization steps, thus the algorithm presented in this study guarantees global optima for the RA problem. Two examples found in the RSM literature are included to illustrate the algorithm, demonstrating its capability of locating better operating conditions than existing computing methods and pointing out particular circumstances (termed the ‘hard case’) where the classical RA procedure fails. An important application to the response modeling problem arising from the philosophy of Taguchi's quality engineering illustrates the hard case. Finally, the utility of the presented TR algorithm is demonstrated through a sequential framework with iterative updates of the TR model under local approximation provided that the predicted response model is a high-order or even non-polynomial function.

In this work, we modify a conjugate gradient (CG) method recently proposed in the literature, where a PRP conjugate gradient method is modified using trust region. Particularly, we propose a hybrid CG method that incorporates the parameters $β^{PRP},$ $β^{FR}$ and $β^{CD},$ and this new search direction satisfies both the trust region feature and the sufficient descent conditions. Furthermore, under suitable conditions the developed method is proved to be globally convergent. The method is tested on some benchmark problems from the literature and numerical results show that it is quite efficient in solving large scale problems.

In this paper, a new nonmonotone adaptive trust region algorithm is proposed for unconstrained optimization by combining a multidimensional filter and the Goldstein-type line search technique. A modified trust region ratio is presented which results in more reasonable consistency between the accurate model and the approximate model. When a trial step is rejected, we use a multidimensional filter to increase the likelihood that the trial step is accepted. If the trial step is still not successful with the filter, a nonmonotone Goldstein-type line search is used in the direction of the rejected trial step. The approximation of the Hessian matrix is updated by the modified Quasi-Newton formula (CBFGS). Under appropriate conditions, the proposed algorithm is globally convergent and superlinearly convergent. The new algorithm shows better performance in terms of the Dolan–Moré performance profile. Numerical results demonstrate the efficiency and robustness of the proposed algorithm for solving unconstrained optimization problems.

The trust region problem, minimization of a quadratic function subject to a spherical trust region constraint, occurs in many optimization algorithms. In a previous paper, the authors introduced an inexpensive approximate solution technique for this problem that involves the solution of a two-dimensional trust region problem. They showed that using this approximation in an unconstrained optimization algorithm leads to the same theoretical global and local convergence properties as are obtained using the exact solution to the trust region problem. This paper reports computational results showing that the two-dimensional minimization approach gives nearly optimal reductions in then-dimension quadratic model over a wide range of test cases. We also show that there is very little difference, in efficiency and reliability, between using the approximate or exact trust region step in solving standard test problems for unconstrained optimization. These results may encourage the application of similar approximate trust region techniques in other contexts.

First-order non-convex Riemannian optimization algorithms have gained recent popularity in structured machine learning problems including principal component analysis and low-rank matrix completion. The current paper presents an efficient Riemannian Stochastic Path Integrated Differential EstimatoR (R-SPIDER) algorithm to solve the finite-sum and online Riemannian non-convex minimization problems. At the core of R-SPIDER is a recursive semi-stochastic gradient estimator that can accurately estimate Riemannian gradient under not only exponential mapping and parallel transport, but also general retraction and vector transport operations. Compared with prior Riemannian algorithms, such a recursive gradient estimation mechanism endows R-SPIDER with lower computational cost in first-order oracle complexity. Specifically, for finite-sum problems with n components, R-SPIDER is proved to converge to an ϵ-approximate stationary point within [Formula: see text] stochastic gradient evaluations, beating the best-known complexity [Formula: see text]; for online optimization, R-SPIDER is shown to converge with [Formula: see text] complexity which is, to the best of our knowledge, the first non-asymptotic result for online Riemannian optimization. For the special case of gradient dominated functions, we further develop a variant of R-SPIDER with improved linear rate of convergence. Extensive experimental results demonstrate the advantage of the proposed algorithms over the state-of-the-art Riemannian non-convex optimization methods.

Inner product preconditioned trust-region methods for frequency-domain full waveform inversion

In this paper, we present two nonmonotone versions of adaptive cubic regularized (ARC) method for unconstrained optimization problems. The proposed methods are a combination of the ARC algorithm with the nonmonotone line search methods introduced by Zhang and Hager [A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14 (2004), pp. 1043–1056] and Ahookhosh et al. [A nonmonotone trust-region line search method for large-scale unconstrained optimization, Appl. Math. Model. 36 (2012), pp. 478–487]. The global convergence analysis for these iterative algorithms is established under suitable conditions. Several numerical examples are given to illustrate the efficiency and robustness of the newly suggested methods. The obtained results show the satisfactory performance of the proposed algorithms when compared to the basic ARC algorithm.

In this paper, a new trust region subproblem is proposed. The trust radius in the new subproblem adjusts itself adaptively. As a result, an adaptive trust region method is constructed based on the new trust region subproblem. The local and global convergence results of the adaptive trust region method are proved. Numerical results indicate that the new method is very efficient.

The complexity of an optimization problem is determined by its decision and objective spaces. Over the past few decades, a large number of works have focused on the performance improvement of metaheuristic algorithms via the objective space, whereas studies related to the decision space have attracted little attentions. Moreover, metaheuristic algorithms may not obtain satisfactory results within an entire feasible region, even if sufficient computational resources are available. Therefore, reducing the search space (i.e., finding a trust region) may be an effective method to ensure that the convergence is sufficiently close to the global optimal region. However, inappropriate subspace size may also weaken the performance of algorithms except for ones with a sufficiently small search space. To alleviate aforementioned problems, a variable search space (VSS) strategy based on a sequential trust region determination approach is proposed in this paper. In the VSS, the entire optimization process is divided into two stages: the first stage is to use an optimization approach for sequentially finding the trust domain of each variable and then determine the best-matched subspace; the second stage is to employ the optimization method for searching an optimal/near-optimal solution within the found trust region. The effectiveness of the VSS is evaluated using two widely used test suites, that is, IEEE CEC2014 and BBOB2012. Experimental results indicate that improving the algorithm performance is an important method for tackling problems, but locating a trust region is also beneficial for metaheuristic algorithms to improve the solution precision, especially for complex optimization problems.