An augmented Lagrangian method for optimization problems with structured geometric constraints

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Dynamic economic dispatch (DED) over multiple time periods is a large-scale coupled spatial-temporal optimization problem. Therefore, the Lagrangian relaxation method has been widely used to split the large-scale optimization problem with coupled structure into several small sub-problems. In order to bring robustness for updating the dual multipliers and yielding convergence without strong assumptions, the augmented Lagrangian relaxation method is introduced in this paper. However, the added penalty term in an augmented Lagrangian function is non-separable, which leads to the difficulty in achieving full decomposition for parallel computation. To address this problem, a diagonal quadratic approximation method is employed to yield an approximated block separation of the non-separable penalty term. Furthermore, the ramp rate constraints are relaxed in this paper, so that the DED model is decomposed into several single-period economic dispatch models that can be efficiently handled in parallel, called the parallel augmented Lagrangian relaxation method. Particularly, the proposed relaxation strategy has a high separability feature which theoretically leads to sound convergence property. Numerical results on the IEEE 118-bus and a practical Polish 2383-bus test system over a different number of time periods show the effectiveness of the proposed method. In addition, the proposed method can be extended to other coupled spatial-temporal scheduling problems in power systems, such as energy storage dispatch.

This paper examines simultaneous design of the network topology and the corresponding edge weights in the presence of a cardinality constraint on the edge set. Network properties of interest for this design problem lead to optimization formulations with convex objectives, convex constraint sets, and cardinality constraints. This class of problems is referred to as the cardinality-constrained optimization problem (CCOP); the cardinality constraint generally makes CCOPs NP-hard. In this paper, a customized alternating direction method of multipliers (ADMM) algorithm aiming to improve the scalability of the solution strategy for large-scale CCOPs is proposed. This algorithm utilizes the special structure of the problem formulation to obtain closed-form solutions during each iterative step of the corresponding ADMM updates. We also provide a convergence proof of the proposed customized ADMM to a stationary point under certain conditions. Simulation results illustrate that the customized ADMM algorithm has a significant computational advantage over existing methods, particularly for large-scale network design problems.

Estimation of Optimal Well Controls Using the Augmented Lagrangian Function with Approximate Derivatives

Stochastic optimization methods have been widely employed to find solutions to structural design optimization problems in the past two decades, especially for truss structures. The primary aim of this study is to introduce a design optimization method combining an augmented Lagrangian function and teaching–learning-based optimization for truss and nontruss structural design optimization. The augmented Lagrangian function serves as a constraint-handling tool in the proposed method and converts a constrained optimization problem into an unconstrained one. On the other hand, teaching–learning-based optimization is employed to resolve the transformed, unconstrained optimization problems. Since the proper values of the Lagrangian multipliers and penalty factors are unknown in advance, the proposed method is implemented in an iterative way to avoid the issue of selecting them, i.e. the Lagrangian multipliers and penalty factors are automatically updated according to the violation level of all constraints. To examine the performance of the proposed method, it is applied on a group of benchmark truss optimization problems and a group of nontruss optimization problems of aircraft wing structures. The computational results obtained by the proposed method are compared to the results produced by both other version of teaching–learning-based optimization and stochastic optimization methods.

Duality in nonlinear programs using augmented Lagrangian functions

An augmented Lagrangian formulation is proposed for large‐slip frictionless contact problems between deformable discretized bodies in two dimensions. Starting from a finite element discretization of the two bodies, a node‐on‐facet element is defined. A non‐linear gap vector and its first variation are derived in terms of the nodal displacements. The relevant action and reaction principle is stated. The gap distance is then related to the conjugate pressure by a (multivalued non‐differentiable) unilateral contact law. The resulting inequality constrained minimization problem is transformed into an unconstrained saddle point problem using an augmented Lagrangian function. Large slip over several facets is possible and the effects of target convexity or concavity are investigated. A generalized Newton method is used to solve the resulting piecewise differentiable equations necessary for equilibrium and contact. The proper tangent (Jacobian) matrices are calculated. The primal (displacements) and dual (contact forces) unknowns are simultaneously updated at each iteration.

We present in this paper new results on the existence of saddle points of augmented Lagrangian functions for constrained nonconvex optimization. Four classes of augmented Lagrangian functions are considered: the essentially quadratic augmented Lagrangian, the exponential-type augmented Lagrangian, the modified barrier augmented Lagrangian, and the penalized exponential-type augmented Lagrangian. We first show that under second-order sufficiency conditions, all these augmented Lagrangian functions possess local saddle points. We then prove that global saddle points of these augmented Lagrangian functions exist under certain mild additional conditions. The results obtained in this paper provide a theoretical foundation for the use of augmented Lagrangians in constrained global optimization. Our findings also give new insights to the role played by augmented Lagrangians in local duality theory of constrained nonconvex optimization.

Mathematical programs with cardinality constraints are optimization problems with an additional constraint which requires the solution to be sparse in the sense that the number of nonzero elements, i.e. the cardinality, is bounded by a given constant. Such programs can be reformulated as a mixed-integer ones in which the sparsity is modeled with the use of complementarity-type constraints. It is shown that the standard relaxation of the integrality leads to a nonlinear optimization program of the striking property that its solutions (global minimizers) are the same as the solutions of the original program with cardinality constraints. Since the number of local minimizers of the relaxed program is typically larger than the number of local minimizers of the cardinality-constrained problem, the relationship between the local minimizers is also discussed in detail. Furthermore, we show under which assumptions the standard KKT conditions are necessary optimality conditions for the relaxed program. The main result obtained for such conditions is significantly different from the existing optimality conditions that are known for the somewhat related class of mathematical programs with complementarity constraints.

Collaborative neurodynamic optimization for solving nonlinear equations

This paper develops and analyzes augmented Lagrangian-based methods for two classes of large-scale optimization problems relevant to modern computational systems: energy-aware network routing with bandwidth allocation and the solution of regularized linear systems. In the first problem, routing and bandwidth allocation are jointly optimized in communication networks under energy constraints, modeled as a mixed-integer nonlinear program. In the second, regularized linear systems are formulated to address ill-posed or illconditioned problems by introducing stabilization terms such as ℓ2 regularization. For both problems, synchronous and asynchronous distributed optimization schemes are designed using decomposition techniques grounded in augmented Lagrangian theory. Extensive numerical experiments across diverse datasets, including network flow instances and benchmark regularized linear systems, demonstrate that the asynchronous variants retain comparable solution quality while significantly improving computational performance, particularly under delay and scalability conditions. These findings reinforce the practical value of asynchronous augmented Lagrangian methods for distributed, high-dimensional, and delay-sensitive optimization problems.

In this paper, we introduce two primal-dual active-set methods for solving large-scale constrained optimization problems. The first method minimizes a sequence of primal-dual augmented Lagrangian functions subject to bounds on the primal variables and artificial bounds on the dual variables. The basic structure is similar to the well-known optimization package Lancelot (Conn, et al. in SIAM J Numer Anal 28:545---572, 1991), which uses the traditional primal augmented Lagrangian function. Like Lancelot, our algorithm may use gradient projection-based methods enhanced by subspace acceleration techniques to solve each subproblem and therefore may be implemented matrix-free. The artificial bounds on the dual variables are a unique feature of our method and serve as a form of dual regularization. Our second algorithm is a two-phase method. The first phase computes iterates using our primal-dual augmented Lagrangian algorithm, which benefits from using cheap gradient projections and matrix-free linear CG calculations. The final iterate produced during this phase is then used as input for phase two, which is a stabilized sequential quadratic programming method (Gill and Robinson in SIAM J Opt 1---45, 2013). Obtaining superlinear local convergence under weak assumptions is an important benefit of the transition to a stabilized sequential quadratic programming algorithm. Interestingly, the bound-constrained subproblem used in phase one is equivalent to the stabilized subproblem used in phase two under certain assumptions. This fact makes our choice of algorithms a natural one.

In this paper, we present a stochastic augmented Lagrangian approach on (possibly infinite-dimensional) Riemannian manifolds to solve stochastic optimization problems with a finite number of deterministic constraints. We investigate the convergence of the method, which is based on a stochastic approximation approach with random stopping combined with an iterative procedure for updating Lagrange multipliers. The algorithm is applied to a multi-shape optimization problem with geometric constraints and demonstrated numerically.

The performance of adaptive beamforming suffers from the imprecise knowledge of the array or signal of interest. Many efforts have been made to develop various beamformers and increase their robustness to these unknown errors. This study proposes a new robust adaptive beamformer based on the alternating direction method of multipliers (ADMM). The proposed method is introduced first as an optimisation problem to estimate the steering vector (SV). Owing to its non-convex property, it is converted to a real-valued condition problem by introducing auxiliary variables. Then, the optimisation problem is reformulated as the augmented Lagrangian function. Finally, the estimated SV can be solved by the ADMM. Numerical results prove the effectiveness of the proposed method.

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs