Augmented Lagrangian and Nonlinear Semidefinite Programs

An augmented Lagrangian approach with a variable transformation in nonlinear programming

ABSTRACTThe proximal method of multipliers was proposed by Rockafellar [Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math Oper Res. 1976;1:97–116] for solving convex programming and it is a kind of proximal point method applied to convex programming. In this paper, we apply this method for solving nonlinear semidefinite programming problems, in which subproblems have better properties than those from the augmented Lagrange method. We prove that, under the linear independence constraint qualification and the strong second-order sufficiency optimality condition, the rate of convergence of the proximal method of multipliers, for a nonlinear semidefinite programming problem, is linear and the ratio constant is proportional to 1/c, where c is the penalty parameter that exceeds a threshold . Moreover, the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to .

Nonlinear semidefinite programming (SDP) problems have received a lot of attentions because of large variety of applications. In this paper, we survey numerical methods for solving nonlinear SDP problems. Three kinds of typical numerical methods are described; augmented Lagrangian methods, sequential SDP methods and primal-dual interior point methods. We describe their typical algorithmic forms and discuss their global and local convergence properties which include rate of convergence.

In this paper, an interior point trust region algorithm for the solution of a class of nonlinear semidefinite programming (SDP) problems is described and analyzed. Such nonlinear and nonconvex programs arise, e.g., in the design of optimal static or reduced order output feedback control laws and have the structure of abstract optimal control problems in a finite dimensional Hilbert space. The algorithm treats the abstract states and controls as independent variables. In particular, an algorithm for minimizing a nonlinear matrix objective functional subject to a nonlinear SDP-condition, a positive definiteness condition, and a nonlinear matrix equation is considered. The algorithm is designed to take advantage of the structure of the problem. It is an extension of an interior point trust region method to nonlinear and nonconvex SDPs, with a special structure which applies sequential quadratic programming techniques to a sequence of barrier problems and uses trust regions to ensure robustness of the iteration. Some convergence results are given, and, finally, several numerical examples demonstrate the applicability of the considered algorithm.

Duality theories in nonlinear semidefinite programming

This paper provides a short introduction to optimization problems with semidefinite constraints. Basic duality and optimality conditions are presented. For linear semidefinite programming some advances by dealing with degeneracy and the semidefinite facial reduction are discussed. Two relatively recent areas of application are presented. Finally a short overview of relevant literature on algorithmic approaches for efficiently solving linear and nonlinear semidefinite programming is provided.

Multiplicative noise (also known as speckle noise) models are central to the study of coherent imaging systems, such as synthetic aperture radar and sonar, and ultrasound and laser imaging. These models introduce two additional layers of difficulties with respect to the standard Gaussian additive noise scenario: (1) the noise is multiplied by (rather than added to) the original image; (2) the noise is not Gaussian, with Rayleigh and Gamma being commonly used densities. These two features of multiplicative noise models preclude the direct application of most state-of-the-art algorithms, which are designed for solving unconstrained optimization problems where the objective has two terms: a quadratic data term (log-likelihood), reflecting the additive and Gaussian nature of the noise, plus a convex (possibly nonsmooth) regularizer (e.g., a total variation or wavelet-based regularizer/prior). In this paper, we address these difficulties by: (1) converting the multiplicative model into an additive one by taking logarithms, as proposed by some other authors; (2) using variable splitting to obtain an equivalent constrained problem; and (3) dealing with this optimization problem using the augmented Lagrangian framework. A set of experiments shows that the proposed method, which we name MIDAL (multiplicative image denoising by augmented Lagrangian), yields state-of-the-art results both in terms of speed and denoising performance.

Compressed sensing (CS) in magnetic resonance imaging (MRI) enables the reconstruction of MR images from highly undersampled k-spaces, and thus substantial reduction of data acquisition time. In this context, edge-preserving and sparsity-promoting regularizers are used to exploit the prior knowledge that MR images are sparse or compressible in a given transform domain and thus to regulate the solution space. In this study, we introduce a new regularization scheme by iterative linearization of the non-convex clipped absolute deviation (SCAD) function in an augmented Lagrangian framework. The performance of the proposed regularization, which turned out to be an iteratively weighted total variation (TV) regularization, was evaluated using 2D phantom simulations and 3D retrospective undersampling of clinical MRI data by different sampling trajectories. It was demonstrated that the proposed regularization technique substantially outperforms conventional TV regularization, especially at reduced sampling rates.

Reduced order output feedback control design for PDE systems using proper orthogonal decomposition and nonlinear semidefinite programming

This paper extends a stability results of nonlinear programming problems by Robinson (Math Program 7:1–16, 1974) to nonlinear semidefinite programming problems. We apply an implicit function theorem to a reformulated Karush–Kuhn–Tucker system, while the Robinson’s work is directly related to the conventional Karush–Kuhn–Tucker system. We show that when the Karush–Kuhn–Tucker condition, constraint nondegeneracy condition, strict complementary condition and second order sufficient condition are satisfied at a feasible point of the original problem, the perturbed problem also satisfies these conditions at some stationary point.

A deterministic global optimization algorithm based on augmented Lagrangian is proposed for solving the nonlinear programming with cone constraints, which is often encountered in technical design and operational research. At each outer iteration, the method requires the -global minimization of the augmented Lagrangian of problems with cone constraints, where. The convergence to an-global minimizer of the original problem is proved. Key words: Global optimization, augmented Lagrangians, cone optimization, algorithms.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

In this paper, we consider a special class of nonlinear semi-definite programming problems that represents the fixed order H2/H¥ synthesis problem. An augmented Lagrangian sequential quadratic programming method combined with a trust region globalization strategy is described, taking advantage of the problem structure and using inexact computations. Some numerical examples that illustrate the performance of the method are given.

An augmented Lagrangian SQP method for solving some special class of nonlinear semi-definite programming problems

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs