An adaptive trust region method and its convergence

A new adaptive trust region algorithm for optimization problems

We consider the problem of large‐scale finite‐sum minimization on Riemannian manifold. We develop a sub‐sampled adaptive trust region method on Riemannian manifolds. Based on inexact information, we adopt adaptive techniques to flexibly adjust the trust region radius in our method. We present the iteration complexity is when the algorithm attains an ‐second‐order stationary point, which matches the result on trust region method. Numerical results for PCA on Grassmann manifold and low‐rank matrix completion are reported to demonstrate the effectiveness of the proposed Riemannian method.

It is well known that among the current methods for unconstrained optimization problems the quasi-Newton methods with global strategy may be the most efficient methods, which have local superlinear convergence. However, when the iterative point is far away from the solution of the problem, quasi-Newton method may proceed slowly for the general unconstrained optimization problems. In this article an adaptive conic trust-region method for unconstrained optimization is presented. Not only the gradient information but also the values of the objective function are used to construct the local model at the current iterative point. Moreover, we define a concept of super steepest descent direction and embed its information into the local model. The amount of computation in each iteration of this adaptive algorithm is the same as that of the standard quasi-Newton method with trust region. Some numerical results show that the modified method requires fewer iterations than the standard methods to reach the solution of the optimization problem. Global and local convergence of the method is also analyzed.

Based on simple quadratic models of the trust region subproblem, we combine the trust region method with the nonmonotone and adaptive techniques to propose a new nonmonotone adaptive trust region algorithm for unconstrained optimization. Unlike traditional trust region method, our trust region subproblem is very simple by using a new scale approximation of the minimizing function’s Hessian. The new method needs less memory capacitance and computational complexity. The convergence results of the method are proved under certain conditions. Numerical results show that the new method is effective and attractive for large scale unconstrained problems.

Optimal control problems are common in aerospace engineering. A Python software program called PySCP is described for solving multiple-phase optimal control problems using sequential convex programming methods. By constructing a series of approximated second-order cone programming subproblems, PySCP approaches to the solution of the original optimal control problem in an iterative way. The key components of the software are described in detail, including convexification, discretization, and the adaptive trust region method. The convexification of the first-order differential dynamic equation is implemented using successive linearization. Six discretization methods, including zero-order hold, first-order hold, Runge-Kutta, and three hp pseudospectral collocation methods, are implemented so that different types of optimal control problems can be tackled efficiently. Adaptive trust region method is employed, and robust convergence is achieved. Both free-final-time problem and fixed-final-time problem can be solved by the software. The application of the software is demonstrated on three optimal control problems with varying complexity. PySCP provides researchers a useful toolkit to solve a wide variety of optimal control problems using sequential convex programming.

An interior point method is proposed for a general nonlinear (non- convex) minimization with linear inequality constraints. This method is a combination of the trust region idea for nonlinearity and affine scaling technique for constraints. Using this method, the original objective function is monotonically decreased.In the proposed approach, a Newton step is derived directly from the complementarity conditions. A trust region subproblem is formed which yields an approximate Newton step as its solution asymptotically. The objective function of the trust region subproblem is the quadratic approximation to the original objective function plus an augmented quadratic convex term. Similar to an augmented Lagrangian function, this augmentation adds positive curvature in the range space of the constraint normals.The global convergence is achieved by possibly using trust regions with different shapes. A reflection technique, which accelerates convergence, is described. Explicit sufficient decrease conditions are proposed.Computational results of a two-dimensional trust region implementation are reported for large-scale problems. Preliminary experiments suggest that this method can be effective a relatively small number of function evaluations are required for some medium and large test problems.

A trust region and affine scaling interior point method (TRAM) is proposed for a general nonlinear minimization with linear inequality constraints in [8]. In the proposed approach, a Newton step is derived from the complementarity conditions. Based on this Newton step, a trust region subproblem is formed, and the original objective function is monotonically decreased. Explicit sufficient decrease conditions are proposed for satisfying complementarity, dual feasibility and second order optimality. The objective of this paper is to establish global and local convergence properties of the proposed trust region and affine scaling interior point method. It is shown that the proposed decrease conditions are sufficient for achieving complementarity, dual feasibility and second order optimality respectively. It is also established that a trust region solution is asymptotically in the interior of the proposed trust region subproblem and a damped trust region step can achieve quadratic convergence.

<p>The Trust Region Subproblem (TRS), which minimizes a nonconvex quadratic function over the unit ball, is an important subproblem in trust region methods for nonlinear optimization. Even though TRS is a nonconvex problem, it can be solved in polynomial time using, for example, a semidefinite programming (SDP) relaxation. Different variants of TRS have been considered from both theoretical and practical perspectives. In this thesis, we study three variants of TRS and their SDP/conic relaxations.</p> <p>We first study an extended trust region subproblem (eTRS) in which the trust region equals the intersection of the unit ball with <em>M</em> linear cuts. When <em>m</em> = 0, when <em>m</em> = 1, or when <em>m</em> = 2 and the linear cuts are parallel, it is known that the eTRS optimal value equals the optimal value of a particular conic relaxation, which is solvable in polynomial time. However, it is also known that, when <em>m</em> ≥2 and at least two of the linear cuts intersect within the ball, i.e., some feasible point of the eTRS satisfies both linear constraints at equality, then the same conic relaxation may admit a gap with eTRS. We show that the conic relaxation admits no gap for arbitrary <em>M</em> as long as the linear cuts are non-intersecting.</p> <p>We then extend our result to a more general setting. We study an eTRS in which a quadratic function is minimized over a structured nonconvex feasible region: the unit ball with <em>M</em> linear cuts and <em>R</em> hollows. In the special case when <em>m</em> = 0 and <em>r</em> = 1, it is known that the eTRS has a tight polynomial-time solvable conic relaxation. We show that a certain conic relaxation is also tight for general <em>R</em> and <em>M</em> as long as the cuts and hollows satisfy some non-intersecting assumptions that generalize the previous paragraph.</p> <p>Finally, intersecting the feasible region of TRS with a second ellipsoid results in the two-trust-region subproblem (TTRS). Even though TTRS can also be solved in polynomial-time, existing approaches do not provide a concise conic relaxation. We investigate the use of conic relaxation for TTRS. Starting from the basic SDP relaxation of TTRS, which admits a gap, recent research has tightened the basic relaxation using valid second-order-cone (SOC) inequalities. For the special case of TTRS in dimension <em>n=2</em>, we fully characterize the remaining valid inequalities, which can be viewed as strengthened versions of the SOC inequalities just mentioned. We also demonstrate that these valid inequalities can be used computationally even when <em>n > 2</em> to solve TTRS instances that were previously unsolved using techniques of conic relaxation.</p>

Hybridizing monotone and nonmonotone approaches, we propose a modified trust region ratio in which more reasonable information is provided about consistency between the exact and the approximate models. Also, we employ an Armijo-type line search strategy to avoid resolving the trust region subproblem whenever a trial step is rejected. We show that the proposed algorithm can preserve global convergence of the traditional trust region algorithm as well as the superlinear convergence property. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr library. They demonstrate practical efficiency of the proposed algorithm in the sense of the Dolan–More performance profile.

In this paper, we propose a new class of adaptive trust region methods for unconstrained optimization problems and develop some convergence properties. In the new algorithms, we use the current iterative information to define a suitable initial trust region radius at each iteration. The initial trust region radius is more reasonable in the sense that the trust region model and the objective function are more consistent at the current iterate. The global convergence, super-linear and quadratic convergence rate are analyzed under some mild conditions. Numerical results show that some special adaptive trust region methods are available and efficient in practical computation.

In this paper, we present a new line search and trust region algorithm for unconstrained optimization problems. The trust region center locates at somewhere in the negative gradient direction with the current best iterative point being on the boundary. By doing these, the trust region subproblems are constructed at a new way different with the traditional ones. Then, we test the efficiency of the new line search and trust region algorithm on some standard benchmarking. The computational results reveal that, for most test problems, the number of function and gradient calculations are reduced significantly.

This thesis is concerned with numerical methods for Mixed-Integer Optimal Control Problems with Combinatorial Constraints. We establish an approximation theorem relating a Mixed-Integer Optimal Control Problem with Combinatorial Constraints to a continuous relaxed convexified Optimal Control Problems with Vanishing Constraints that provides the basis for numerical computations. We develop a a Vanishing- Constraint respecting rounding algorithm to exploit this correspondence computationally. Direct Discretization of the Optimal Control Problem with Vanishing Constraints yield a subclass of Mathematical Programs with Equilibrium Constraints. Mathematical Programs with Equilibrium Constraint constitute a class of challenging problems due to their inherent non-convexity and non-smoothness. We develop an active-set algorithm for Mathematical Programs with Equilibrium Constraints and prove global convergence to Bouligand stationary points of this algorithm under suitable technical conditions. For efficient computation of Newton-type steps of Optimal Control Problems, we establish the Generalized Lanczos Method for trust region problems in a Hilbert space context. To ensure real-time feasibility in Online Optimal Control Applications with tracking-type Lagrangian objective, we develop a Gaus-Newton preconditioner for the iterative solution method of the trust region problem. We implement the proposed methods and demonstrate their applicability and efficacy on several benchmark problems.

A trust-region algorithm combining line search filter method with Lagrange merit function for nonlinear constrained optimization

Limited-memory quasi-Newton methods and trust-region methods represent two efficient approaches used for solving unconstrained optimization problems. A straightforward combination of them deteriorates the efficiency of the former approach, especially in the case of large-scale problems. For this reason, the limited-memory methods are usually combined with a line search. We show how to efficiently combine limited-memory and trust-region techniques. One of our approaches is based on the eigenvalue decomposition of the limited-memory quasi-Newton approximation of the Hessian matrix. The decomposition allows for finding a nearly-exact solution to the trust-region subproblem defined by the Euclidean norm with an insignificant computational overhead as compared with the cost of computing the quasi-Newton direction in line-search limited-memory methods. The other approach is based on two new eigenvalue-based norms. The advantage of the new norms is that the trust-region subproblem is separable and each of the smaller subproblems is easy to solve. We show that our eigenvalue-based limited-memory trust-region methods are globally convergent. Moreover, we propose improved versions of the existing limited-memory trust-region algorithms. The presented results of numerical experiments demonstrate the efficiency of our approach which is competitive with line-search versions of the L-BFGS method.

Fiber optical gyroscopes (FOGs) have been applied widely in navigation for mobile robots. The precision of outputs from FOGs is affected significantly by uncertain disturbances due to road conditions in practice. It is urgent to eliminate uncertain disturbances from outputs of sensors in harsh environments. This paper proposes an adaptive method of signal processing base on wavelet and fuzzy logic for outdoor mobile robots in different road condition. Property of disturbances in different outdoor road is taken into account to determine filtering thresholds in this research. Corresponding characteristics of real-time data were processed by a fuzzy logic inference system when mobile robots moved, and then they were de-noising by means of wavelet technique. It was proved that this proposed adaptive method can eliminate disturbances due to road irregularities and improve accuracy of surrogate data by experiments.