On Saddle Points of Augmented Lagrangians for Constrained Nonconvex Optimization

In this article, we introduce a unified class of augmented Lagrangian functions for constrained non-convex optimization problems which include many types of the augmented Lagrangians. We first get the zero duality gap property between the primal problem and the augmented Lagrangian dual problem. Then, under second-order sufficiency conditions, we prove that this class of augmented Lagrangian functions possesses local saddle points. Finally, we show the existence of global saddle points without requiring the compactness of X and the uniqueness of the global solution.

We establish local convergence and rate of convergence of the classical augmented Lagrangian algorithm under the sole assumption that the dual starting point is close to a multiplier satisfying the second-order sufficient optimality condition. In particular, no constraint qualifications of any kind are needed. Previous literature on the subject required, in addition, the linear independence constraint qualification and either the strict complementarity assumption or a stronger version of the second-order sufficient condition. That said, the classical results allow the initial multiplier estimate to be far from the optimal one, at the expense of proportionally increasing the threshold value for the penalty parameters. Although our primary goal is to avoid constraint qualifications, if the stronger assumptions are introduced, then starting points far from the optimal multiplier are allowed within our analysis as well. Using only the second-order sufficient optimality condition, for penalty parameters large enough we prove primal-dual Q-linear convergence rate, which becomes superlinear if the parameters are allowed to go to infinity. Both exact and inexact solutions of subproblems are considered. In the exact case, we further show that the primal convergence rate is of the same Q-order as the primal-dual rate. Previous assertions for the primal sequence all had to do with the weaker R-rate of convergence and required the stronger assumptions cited above. Finally, we show that under our assumptions one of the popular rules of controlling the penalty parameters ensures their boundedness.

In this paper, we develop the sufficient conditions for the existence of local and global saddle points of two classes of augmented Lagrangian functions for nonconvex optimization problem with both equality and inequality constraints, which improve the corresponding results in available papers. The main feature of our sufficient condition for the existence of global saddle points is that we do not need the uniqueness of the optimal solution. Furthermore, we show that the existence of global saddle points is a necessary and sufficient condition for the exact penalty representation in the framework of augmented Lagrangians. Based on these, we convert a class of generalized semi-infinite programming problems into standard semi-infinite programming problems via augmented Lagrangians. Some new first-order optimality conditions are also discussed.

Duality in nonlinear programs using augmented Lagrangian functions

Estimation of Optimal Well Controls Using the Augmented Lagrangian Function with Approximate Derivatives

An augmented Lagrangian formulation is proposed for large‐slip frictionless contact problems between deformable discretized bodies in two dimensions. Starting from a finite element discretization of the two bodies, a node‐on‐facet element is defined. A non‐linear gap vector and its first variation are derived in terms of the nodal displacements. The relevant action and reaction principle is stated. The gap distance is then related to the conjugate pressure by a (multivalued non‐differentiable) unilateral contact law. The resulting inequality constrained minimization problem is transformed into an unconstrained saddle point problem using an augmented Lagrangian function. Large slip over several facets is possible and the effects of target convexity or concavity are investigated. A generalized Newton method is used to solve the resulting piecewise differentiable equations necessary for equilibrium and contact. The proper tangent (Jacobian) matrices are calculated. The primal (displacements) and dual (contact forces) unknowns are simultaneously updated at each iteration.

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs

In this paper, a new augmented Lagrangian penalty function for constrained optimization problems is studied. The dual properties of the augmented Lagrangian objective penalty function for constrained optimization problems are proved. Under some conditions, the saddle point of the augmented Lagrangian objective penalty function satisfies the first-order Karush-Kuhn-Tucker (KKT) condition. Especially, when the KKT condition holds for convex programming its saddle point exists. Based on the augmented Lagrangian objective penalty function, an algorithm is developed for finding a global solution to an inequality constrained optimization problem and its global convergence is also proved under some conditions.

This paper is devoted to the theoretical and numerical investigation of an augmented Lagrangian method for the solution of optimization problems with geometric constraints. Specifically, we study situations where parts of the constraints are nonconvex and possibly complicated, but allow for a fast computation of projections onto this nonconvex set. Typical problem classes which satisfy this requirement are optimization problems with disjunctive constraints (like complementarity or cardinality constraints) as well as optimization problems over sets of matrices which have to satisfy additional rank constraints. The key idea behind our method is to keep these complicated constraints explicitly in the constraints and to penalize only the remaining constraints by an augmented Lagrangian function. The resulting subproblems are then solved with the aid of a problem-tailored nonmonotone projected gradient method. The corresponding convergence theory allows for an inexact solution of these subproblems. Nevertheless, the overall algorithm computes so-called Mordukhovich-stationary points of the original problem under a mild asymptotic regularity condition, which is generally weaker than most of the respective available problem-tailored constraint qualifications. Extensive numerical experiments addressing complementarity- and cardinality-constrained optimization problems as well as a semidefinite reformulation of MAXCUT problems visualize the power of our approach.

Stochastic optimization methods have been widely employed to find solutions to structural design optimization problems in the past two decades, especially for truss structures. The primary aim of this study is to introduce a design optimization method combining an augmented Lagrangian function and teaching–learning-based optimization for truss and nontruss structural design optimization. The augmented Lagrangian function serves as a constraint-handling tool in the proposed method and converts a constrained optimization problem into an unconstrained one. On the other hand, teaching–learning-based optimization is employed to resolve the transformed, unconstrained optimization problems. Since the proper values of the Lagrangian multipliers and penalty factors are unknown in advance, the proposed method is implemented in an iterative way to avoid the issue of selecting them, i.e. the Lagrangian multipliers and penalty factors are automatically updated according to the violation level of all constraints. To examine the performance of the proposed method, it is applied on a group of benchmark truss optimization problems and a group of nontruss optimization problems of aircraft wing structures. The computational results obtained by the proposed method are compared to the results produced by both other version of teaching–learning-based optimization and stochastic optimization methods.

The paper describes a new approach to the optimal-power-flow problem based on Newton's method which it operates with an augmented Lagrangian function associated with the original problem. The function aggregates all the equality and inequality constraints. The first-order necessary conditions for optimality are reached by Newton's method, and by updating the dual variables and the penalty terms associated with the inequality constraints. The proposed approach does not have to identify the set of binding constraints and can be utilised for an infeasible starting point. The sparsity of the Hessian matrix of the augmented Lagrangian is completely exploited in the computational implementation. Tests results are presented to show the good performance of this approach.

In this paper, we introduce two primal-dual active-set methods for solving large-scale constrained optimization problems. The first method minimizes a sequence of primal-dual augmented Lagrangian functions subject to bounds on the primal variables and artificial bounds on the dual variables. The basic structure is similar to the well-known optimization package Lancelot (Conn, et al. in SIAM J Numer Anal 28:545---572, 1991), which uses the traditional primal augmented Lagrangian function. Like Lancelot, our algorithm may use gradient projection-based methods enhanced by subspace acceleration techniques to solve each subproblem and therefore may be implemented matrix-free. The artificial bounds on the dual variables are a unique feature of our method and serve as a form of dual regularization. Our second algorithm is a two-phase method. The first phase computes iterates using our primal-dual augmented Lagrangian algorithm, which benefits from using cheap gradient projections and matrix-free linear CG calculations. The final iterate produced during this phase is then used as input for phase two, which is a stabilized sequential quadratic programming method (Gill and Robinson in SIAM J Opt 1---45, 2013). Obtaining superlinear local convergence under weak assumptions is an important benefit of the transition to a stabilized sequential quadratic programming algorithm. Interestingly, the bound-constrained subproblem used in phase one is equivalent to the stabilized subproblem used in phase two under certain assumptions. This fact makes our choice of algorithms a natural one.

Dynamic economic dispatch (DED) over multiple time periods is a large-scale coupled spatial-temporal optimization problem. Therefore, the Lagrangian relaxation method has been widely used to split the large-scale optimization problem with coupled structure into several small sub-problems. In order to bring robustness for updating the dual multipliers and yielding convergence without strong assumptions, the augmented Lagrangian relaxation method is introduced in this paper. However, the added penalty term in an augmented Lagrangian function is non-separable, which leads to the difficulty in achieving full decomposition for parallel computation. To address this problem, a diagonal quadratic approximation method is employed to yield an approximated block separation of the non-separable penalty term. Furthermore, the ramp rate constraints are relaxed in this paper, so that the DED model is decomposed into several single-period economic dispatch models that can be efficiently handled in parallel, called the parallel augmented Lagrangian relaxation method. Particularly, the proposed relaxation strategy has a high separability feature which theoretically leads to sound convergence property. Numerical results on the IEEE 118-bus and a practical Polish 2383-bus test system over a different number of time periods show the effectiveness of the proposed method. In addition, the proposed method can be extended to other coupled spatial-temporal scheduling problems in power systems, such as energy storage dispatch.

This paper treats a distributed optimal control problem for a tumor growth model of viscous Cahn--Hilliard type. The evolution of the tumor fraction is governed by a thermodynamic force induced by a double-well potential of logarithmic type. The cost functional contains a nondifferentiable term in order to enhance the occurrence of sparsity effects in the optimal controls, i.e., of subdomains of the space-time cylinder where the controls vanish. In the context of cancer therapies, sparsity is very important in order that the patient is not exposed to unnecessary intensive medical treatment. In this work, we focus on the derivation of second-order sufficient optimality conditions for the optimal control problem. While in previous works on the system under investigation such conditions have been established for the case without sparsity, the case with sparsity has not been treated before. The results obtained in this paper also improve the known results on this phase field model for the case without sparsity.

Constrained optimization is central to large-scale machine learning, particularly in parallel and distributed environments. This paper presents a comprehensive study of augmented Lagrangian–based algorithms for such problems, including classical Lagrangian relaxation, the method of multipliers, the Alternating Direction Method of Multipliers (ADMM), Bertsekas’ algorithm, Tatjewski’s method, and the Separable Augmented Lagrangian Algorithm (SALA). We develop a unified theoretical framework, analyze convergence properties and decomposition strategies, and evaluate these methods on two representative classes of tasks: regularized linear systems and K-means clustering. Numerical experiments on synthetic and real-world datasets show that Bertsekas’ method consistently achieves the best balance of convergence speed and solution quality, while ADMM offers practical scalability under decomposition but struggles in high-dimensional or ill-conditioned settings. Tatjewski’s method benefits significantly from partitioning, whereas the classical Augmented Lagrangian approach proves computationally inefficient for large-scale problems. These findings clarify the trade-offs among augmented Lagrangian algorithms, highlighting Bertsekas’ method as the most effective for distributed optimization and providing guidance for algorithm selection in large-scale machine learning applications.