An augmented Lagrangian method of optimization problems with cone constraints

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

During the last three decades, the research area of systems engineering has emerged as a domain of fundamental importance and major impact within chemical engineering, as well as a cornerstone area in interdisciplinary research initiatives with computer science, operations research, applied mathematics, materials and life sciences. This is attributed to the unique characteristics of systems engineering which are the combination of analysis and synthesis for the design, optimization, and operation of processes and products. The product and process discovery research efforts are founded on fundamental advances in mathematical modeling, optimization theory and algorithms, and insights derived either from existing operations and/or from biology, chemistry, and physics. The fundamental advances are epitomized through new theoretical, algorithmic, and modeling frameworks for (a) mixed-integer linear and nonlinear optimization, (b) deterministic global optimization, and (c) dynamic simulation and optimization. The proposed modeling and optimization approaches have multiscale applications ranging from macroscopic to mesoscopic to microscopic systems, and which provide a natural and fundamental link between systems engineering, computational chemistry, computational biology and systems biology. Approaches based on mixed-integer linear and nonlinear optimization which were typically identified with process synthesis, scheduling and planning applications, have entered the domains of gene regulatory networks, metabolic networks, signal transduction networks, beta-sheet topology prediction in proteins, de novo peptide and protein design, DNA recombination, phase problem in X-ray crystallography, side chain optimization in protein prediction, peptide identification via tandem mass spectroscopy. Approaches based on deterministic global optimization associated with process design, synthesis, scheduling, and pooling/blending applications, are now in the main stream of product design, structure prediction in protein folding, dynamics of protein folding, NMR protein structure refinement, and de novo protein design. Approaches based on dynamic models and large scale optimization which were identified with process models and their applications, are suited for metabolic and signal transduction networks. As a result, a synergism between systems engineering, computational biology, and systems biology has evolved gradually, opened new research avenues and has reached the stage where fascinating research contributions address important questions in computational biology with methods and tools from systems engineering which combine mathematical rigor with key biological insights. This article provides a perspective on the challenges and opportunities that emerge from the fundamental developments in the research fields of systems engineering and computational biology. The advances in the areas of deterministic global optimization and process scheduling are introduced first, followed by their respective research opportunities. The revolution of genomics is discussed next with a particular focus on the advances and challenges in the areas of structure prediction in protein folding, de novo peptide and protein design, and peptide and protein identification via tandem mass spectroscopy. This article is based on material presented as an invited talk at the session on “The Future of Chemical Engineering Research III” during the 2004 annual AIChE meeting.

ABSTRACTThe proximal method of multipliers was proposed by Rockafellar [Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math Oper Res. 1976;1:97–116] for solving convex programming and it is a kind of proximal point method applied to convex programming. In this paper, we apply this method for solving nonlinear semidefinite programming problems, in which subproblems have better properties than those from the augmented Lagrange method. We prove that, under the linear independence constraint qualification and the strong second-order sufficiency optimality condition, the rate of convergence of the proximal method of multipliers, for a nonlinear semidefinite programming problem, is linear and the ratio constant is proportional to 1/c, where c is the penalty parameter that exceeds a threshold . Moreover, the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to .

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs

Obtaining guaranteed lower bounds for problems with unknown algebraic form has been a major challenge in derivative-free optimization. In this work, we present a deterministic global optimization method for black-box problems where the derivatives are not available or it is computationally expensive to obtain. However, a global upper bound on the diagonal Hessian elements is known. An edge-concave underestimator (Hasan in J Glob Optim 71:735–752, 2018) can be then constructed with vertex polyhedral solution. Evaluating this underestimator only at the vertices leads to a valid lower bound on the original black-box problem. We have implemented this lower bounding technique within a branch-and-bound framework and assessed its computational performance in locating $$\epsilon $$-global optimal solution for several box-constrained, nonconvex black-box functions.

It is sometimes necessary to consider more general nonlinear programs than the ones we have already studied: linear, quadratic, or conic programs. We give a brief introduction to this vast topic, and we discuss an application to estimating a volatility surface. Nonlinear Programming Consider a very general optimization problem of the form or the equivalent more concise form where. In the special case when all functions , hi are linear, problem (20.1) is a linear program as discussed in Chapter 2. When some of the functions , are nonlinear, problem (20.1) is a nonlinear program . Many practical problems are naturally formulated as nonlinear programs. We already saw quadratic programming and conic programming in earlier chapters. However, the family of problems that can be formulated as nonlinear programs is enormous, as many, if not most, imaginable kinds of constraints and objectives can be cast in terms of nonlinear functions. (For some noteworthy examples, see the exercises at the end of the chapter.) The immense modeling power of nonlinear programming comes at a cost: unlike linear, quadratic, and conic programming, which have a solid theory and are solvable via a few algorithmic templates, both the theory and methods to solve general nonlinear programs are far more complicated. Different types of nonlinear programs, determined by structural properties of the objective and constraint functions, are amenable to different types of algorithms. The subsequent sections sketch the main theory and most popular algorithmic ideas. A comprehensive treatment of this vast topic is beyond the scope of this textbook. We refer the reader to the excellent references Bertsekas (1999), Guler (2010), and Nocedal and Wright (2006) for more details. Numerical Nonlinear Programming Solvers There are numerous software packages for solving nonlinear programs. The following are some popular ones. We list them according to the class of algorithms (discussed in Section 20.4) that they are based on: (1) CONOPT, GRG2, Excel SOLVER. These solvers are based on the generalized reduced-gradient method . (2) MATLAB optimization toolbox, SNOPT, NLPQL. These solvers are based on sequential quadratic programming. (3) MINOS, LANCELOT. These solvers are based on an augmented Lagrangian approach. (4) MOSEK, LOQO, IPOPT. These solvers are based on interior-point methods.

This chapter introduces two very important concepts in constrained nonlinear optimization. These are penalty and augmented Lagrangian concepts. The idea is that both these concepts replace the original problem by a sequence of sub-problems in which the constraints are expressed by terms added to the objective function. The penalty concept is implemented in two different methods. The quadratic penalty method adds to the objective function a multiple of the square of the violation of each constraint and solves a sequence of unconstrained optimization sub-problems. Simple and enough intuitive, this approach has some important deficiencies. The nonsmooth exact penalty method, on the other hand, solves a single unconstrained optimization problem. In this approach, a popular function is the l1 penalty function. The problem with this method is that the nonsmoothness may create complications in numerical implementations. Finally, the second concept is the multiplier method or the augmented Lagrangian method, which explicitly uses Lagrange multiplier estimates in order to avoid the ill-conditioning of the quadratic penalty method.

A hybrid differential evolution augmented Lagrangian method for constrained numerical and engineering optimization

Contributors

We consider here the average deviation as the most important objective when designing numerical techniques and algorithms. We call that a Bayesian approach. We start by describing the Bayesian approach to the continu­ ous global optimization. Then we show how to apply the results to the adaptation of parameters of randomized techniques of optimiza­ tion. We assume that there exists a simple function which roughly predicts the consequences of decisions. We call it heuristics. We define the probability of a decision by a randomized decision func­ tion depending on heuristics. We fix this decision function, except for some parameters that we call the decision parameters. We repeat the randomized decision procedure several times given the decision parameters and regard the best outcome as a result. We optimize the decision parameters to make the search more efficient. Thus we replace the original optimization problem by an auxiliary problem of continuous stochastic optimization. We solve the auxiliary problem by the Bayesian methods of global op­ timization. Therefore we call the approach as the Bayesian one. We discuss the advantages and disadvantages of the Bayesian approach. We describe the applications to some of discrete pro-

Augmented Lagrangian methods are effective tools for solving large-scale nonlinear programming problems. At each outer iteration, a minimization subproblem with simple constraints, whose objective function depends on updated Lagrange multipliers and penalty parameters, is approximately solved. When the penalty parameter becomes very large, solving the subproblem becomes difficult; therefore, the effectiveness of this approach is associated with the boundedness of the penalty parameters. In this paper, it is proved that under more natural assumptions than the ones employed until now, penalty parameters are bounded. For proving the new boundedness result, the original algorithm has been slightly modified. Numerical consequences of the modifications are discussed and computational experiments are presented.

Due to the possible lack of primal-dual-type error bounds, it was not clear whether the Karush–Kuhn–Tucker (KKT) residuals of the sequence generated by the augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) problems converge superlinearly. In this paper, we resolve this issue by establishing the R-superlinear convergence of the KKT residuals generated by the ALM under only a mild quadratic growth condition on the dual of CCCP, with easy-to-implement stopping criteria for the augmented Lagrangian subproblems. This discovery may help to explain the good numerical performance of several recently developed semismooth Newton-CG based ALM solvers for linear and convex quadratic semidefinite programming.

In this paper, we consider a special class of nonlinear semi-definite programming problems that represents the fixed order H2/H¥ synthesis problem. An augmented Lagrangian sequential quadratic programming method combined with a trust region globalization strategy is described, taking advantage of the problem structure and using inexact computations. Some numerical examples that illustrate the performance of the method are given.

An augmented Lagrangian SQP method for solving some special class of nonlinear semi-definite programming problems

Hammerstein–Wiener models constitute a significant class of block-structured dynamic models, as they approximate process nonlinearities on the basis of input–output data without requiring identification of a full nonlinear process model. Optimization problems with Hammerstein–Wiener models embedded are nonconvex, and thus local optimization methods may obtain suboptimal solutions. In this work, we develop a deterministic global optimization strategy that exploits the specific structure of Hammerstein–Wiener models to extend existing theory on global optimization of systems with linear dynamics. At first, we discuss alternative formulations of the dynamic optimization problem with Hammerstein–Wiener models embedded, demonstrating that careful selection of the optimization variables of the problem can offer significant numerical advantages to the solution approach. Then, we develop convex relaxations for the proposed optimization problem and discuss implementation aspects to obtain the global solution focusing on a control parametrization technique. Finally, we apply our optimization strategy to case studies comprising both offline and online dynamic optimization problems. The results confirm an improved computational performance of the proposed solution approach over alternative options not exploiting the linear dynamics for all considered examples. They also underline the tractability of deterministic global dynamic optimization when using few control intervals in online applications like nonlinear model predictive control.