An augmented Lagrangian approach with a variable transformation in nonlinear programming

In this paper, we introduce an augmented Lagrangian for nonlinear semidefinite programs. Some basic properties of the augmented Lagrangian such as differentiabilty, monotonicity and convexity, are discussed. Necessary and sufficient conditions for a strong duality property and an exact penalty representation in the framework of augmented Lagrangian are derived. Under certain conditions, it is shown that any limit point of a sequence of stationary points of augmented Lagrangian problems is a Karuh, Kuhn-Tucker (for short, KKT) point of the original semidefinite program.

It is sometimes necessary to consider more general nonlinear programs than the ones we have already studied: linear, quadratic, or conic programs. We give a brief introduction to this vast topic, and we discuss an application to estimating a volatility surface. Nonlinear Programming Consider a very general optimization problem of the form or the equivalent more concise form where. In the special case when all functions , hi are linear, problem (20.1) is a linear program as discussed in Chapter 2. When some of the functions , are nonlinear, problem (20.1) is a nonlinear program . Many practical problems are naturally formulated as nonlinear programs. We already saw quadratic programming and conic programming in earlier chapters. However, the family of problems that can be formulated as nonlinear programs is enormous, as many, if not most, imaginable kinds of constraints and objectives can be cast in terms of nonlinear functions. (For some noteworthy examples, see the exercises at the end of the chapter.) The immense modeling power of nonlinear programming comes at a cost: unlike linear, quadratic, and conic programming, which have a solid theory and are solvable via a few algorithmic templates, both the theory and methods to solve general nonlinear programs are far more complicated. Different types of nonlinear programs, determined by structural properties of the objective and constraint functions, are amenable to different types of algorithms. The subsequent sections sketch the main theory and most popular algorithmic ideas. A comprehensive treatment of this vast topic is beyond the scope of this textbook. We refer the reader to the excellent references Bertsekas (1999), Guler (2010), and Nocedal and Wright (2006) for more details. Numerical Nonlinear Programming Solvers There are numerous software packages for solving nonlinear programs. The following are some popular ones. We list them according to the class of algorithms (discussed in Section 20.4) that they are based on: (1) CONOPT, GRG2, Excel SOLVER. These solvers are based on the generalized reduced-gradient method . (2) MATLAB optimization toolbox, SNOPT, NLPQL. These solvers are based on sequential quadratic programming. (3) MINOS, LANCELOT. These solvers are based on an augmented Lagrangian approach. (4) MOSEK, LOQO, IPOPT. These solvers are based on interior-point methods.

This paper establishes the relations between the changes in the values of the augmented multipliers and the changes in the values of the constraint and objective functions for a non-linear, non-convex programme based on the augmented Lagrangian which was introduced by one of the present authors. If the penalty parameters are sufficiently large, the monotonic relations hold between the augmented multipliers and the constraint functions near the optimal solution. Illustrative examples show that the classical Lagrangian fails to give rise to the relations for a. non-convex programme.

Bioluminescence imaging (BLI) is an optical molecular imaging modality for monitoring physiological and pathological activities at the molecular level. The information of bioluminescent probe distribution in small animals can be threedimensionally and quantitatively obtained by bioluminescence tomography (BLT). Due to ill-posed nature, BLT may bear multiple solutions and aberrant reconstruction in the presence of measurement noise and optical parameter mismatches. Among different regularization methods, L2-type regularization strategy is the most popular and commonly-applied method, which minimizes the output-least-square formulation incorporated with the l2-norm regularization term to stabilize the problem. However, it often imposes over-smoothing on the reconstruction results. In contrast, for many practical applications, such as early detection of tumors, the volumes of the bioluminescent sources are very small compared with the whole body. In this paper, L1 regularization is used to fully take advantage of the sparsity prior knowledge and improve both efficiency and stability. And then a reconstruction method based on the augmented Lagrangian approach is proposed, which considers the BLT problem as the constrained optimization problem and employs the Bregman iterative method to deal with it. By using "divide and conquer" approach, the optimization problem can be exactly and fast solved by iteratively solving a sequence of unconstrained subproblems. To evaluate the performance of the proposed method in turbid mouse geometry, stimulate experiments with a heterogeneous 3D mouse atlas are conducted. In addition, physical experiments further demonstrate the potential of the proposed algorithm in practical applications.

In this paper, we investigate a non-elitist Evolution Strategy designed to handle black-box constraints by an adaptive Augmented Lagrangian penalty approach, AL-(μ/μw, λ)-CMA-ES, on problems with up to 28 constraints. Based on stability and performance observations, we propose an improved default parameter setting. We exhibit failure cases of the Augmented Lagrangian technique and show how surrogate modeling of the constraints can overcome some difficulties. Several variants of AL-CMA-ES are compared on a set of nonlinear constrained problems from the literature. Simple adaptive penalty techniques serve as a baseline for comparison.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

A deterministic global optimization algorithm based on augmented Lagrangian is proposed for solving the nonlinear programming with cone constraints, which is often encountered in technical design and operational research. At each outer iteration, the method requires the -global minimization of the augmented Lagrangian of problems with cone constraints, where. The convergence to an-global minimizer of the original problem is proved. Key words: Global optimization, augmented Lagrangians, cone optimization, algorithms.

In this paper, we develop the sufficient conditions for the existence of local and global saddle points of two classes of augmented Lagrangian functions for nonconvex optimization problem with both equality and inequality constraints, which improve the corresponding results in available papers. The main feature of our sufficient condition for the existence of global saddle points is that we do not need the uniqueness of the optimal solution. Furthermore, we show that the existence of global saddle points is a necessary and sufficient condition for the exact penalty representation in the framework of augmented Lagrangians. Based on these, we convert a class of generalized semi-infinite programming problems into standard semi-infinite programming problems via augmented Lagrangians. Some new first-order optimality conditions are also discussed.

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs

Constrained optimization is central to large-scale machine learning, particularly in parallel and distributed environments. This paper presents a comprehensive study of augmented Lagrangian–based algorithms for such problems, including classical Lagrangian relaxation, the method of multipliers, the Alternating Direction Method of Multipliers (ADMM), Bertsekas’ algorithm, Tatjewski’s method, and the Separable Augmented Lagrangian Algorithm (SALA). We develop a unified theoretical framework, analyze convergence properties and decomposition strategies, and evaluate these methods on two representative classes of tasks: regularized linear systems and K-means clustering. Numerical experiments on synthetic and real-world datasets show that Bertsekas’ method consistently achieves the best balance of convergence speed and solution quality, while ADMM offers practical scalability under decomposition but struggles in high-dimensional or ill-conditioned settings. Tatjewski’s method benefits significantly from partitioning, whereas the classical Augmented Lagrangian approach proves computationally inefficient for large-scale problems. These findings clarify the trade-offs among augmented Lagrangian algorithms, highlighting Bertsekas’ method as the most effective for distributed optimization and providing guidance for algorithm selection in large-scale machine learning applications.

Duality in nonlinear programs using augmented Lagrangian functions

A mixed penalty-duality formulation of the frictional contact problem, inspired from an augmented Lagrangian approach is proposed. The continuity of the resulting conewise linear operator is used to establish a uniqueness condition on the coefficient of friction. Modified and generalized Newton methods are examined and sufficient conditions for their convergence conjectured. A cylindrical frictional contact problem assesses the stability of the method. Mixed penalty-duality methods are found more accurate and stabler than penalty methods and as economical as them.

We present in this paper new results on the existence of saddle points of augmented Lagrangian functions for constrained nonconvex optimization. Four classes of augmented Lagrangian functions are considered: the essentially quadratic augmented Lagrangian, the exponential-type augmented Lagrangian, the modified barrier augmented Lagrangian, and the penalized exponential-type augmented Lagrangian. We first show that under second-order sufficiency conditions, all these augmented Lagrangian functions possess local saddle points. We then prove that global saddle points of these augmented Lagrangian functions exist under certain mild additional conditions. The results obtained in this paper provide a theoretical foundation for the use of augmented Lagrangians in constrained global optimization. Our findings also give new insights to the role played by augmented Lagrangians in local duality theory of constrained nonconvex optimization.

In this paper we consider a nonmonotone (mixed) variational inequality (VI) model with (nonlinear) convex conic constraints. Through developing an equivalent Lagrangian function-like primal-dual saddle point system for the VI model in question, we introduce an augmented Lagrangian primal-dual method, called ALAVI (Augmented Lagrangian Approach to Variational Inequality) in the paper, for solving a general constrained VI model. Under an assumption, called the primal-dual variational coherence condition in the paper, we prove the convergence of ALAVI. Next, we show that many existing generalized monotonicity properties are sufficient—though by no means necessary—to imply the abovementioned coherence condition and thus are sufficient to ensure convergence of ALAVI. Under that assumption, we further show that ALAVI has in fact an [Formula: see text] global rate of convergence where k is the iteration count. By introducing a new gap function, this rate further improves to be [Formula: see text] if the mapping is monotone. Finally, we show that under a metric subregularity condition, even if the VI model may be nonmonotone, the local convergence rate of ALAVI improves to be linear. Numerical experiments on some randomly generated highly nonlinear and nonmonotone VI problems show the practical efficacy of the newly proposed method. Funding: L. Zhao and D. Zhu were partially supported by the Major Project of the National Natural Science Foundation of China [Grant 72293582], the National Key R&D Program of China [Grant 2023YFA0915202], and the Fundamental Research Funds for the Central Universities (the Interdisciplinary Program of Shanghai Jiao Tong University) [Grant YG2024QNA36]. L. Zhao was partially supported by the Startup Fund for Young Faculty at SJTU (SFYF at SJTU) [Grant 22X010503839].

Compressed sensing (CS) in magnetic resonance imaging (MRI) enables the reconstruction of MR images from highly undersampled k-spaces, and thus substantial reduction of data acquisition time. In this context, edge-preserving and sparsity-promoting regularizers are used to exploit the prior knowledge that MR images are sparse or compressible in a given transform domain and thus to regulate the solution space. In this study, we introduce a new regularization scheme by iterative linearization of the non-convex clipped absolute deviation (SCAD) function in an augmented Lagrangian framework. The performance of the proposed regularization, which turned out to be an iteratively weighted total variation (TV) regularization, was evaluated using 2D phantom simulations and 3D retrospective undersampling of clinical MRI data by different sampling trajectories. It was demonstrated that the proposed regularization technique substantially outperforms conventional TV regularization, especially at reduced sampling rates.