A nonlinear Lagrangian for constrained optimization problems

This paper establishes a theory framework of a class of nonlinear Lagrangians for solving nonlinear programming problems with inequality constraints. A set of conditions are proposed to guarantee the convergence of nonlinear Lagrangian algorithms, to analyze condition numbers of nonlinear Lagrangian Hessians as well as to develop the dual approaches. These conditions are satisfied by well-known nonlinear Lagrangians appearing in literature. The convergence theorem shows that the dual algorithm based on any nonlinear Lagrangian in the class is locally convergent when the penalty parameter is less than a threshold under a set of suitable conditions on problem functions and the error bound solution, depending on the penalty parameter, is also established. The paper also develops the dual problems based on the proposed nonlinear Lagrangians, and the related duality theorem and saddle point theorem are demonstrated. Furthermore, it is shown that the condition numbers of Lagrangian Hessians at optimal solutions are proportional to the controlling penalty parameters. We report some numerical results obtained by using nonlinear Lagrangians.

A nonlinear augmented Lagrangian for constrained minimax problems

This paper studies the properties of a class of nonlinear Lagrangians for nonlinear programming with inequality constraints. It's shown that under a set of conditions this class of Lagrange algorithm is locally convergent when the penalty parameter is larger than a threshold. An error bound estimate of the solution, depending on the penalty, is also established. The paper also discusses the properties of the dual function associated with the proposed nonlinear Lagrangians. Finally, the dual algorithm corresponding to the proposed nonlinear Lagrangians is developed and used to solve some numerical examples by using the nonlinear Lagrangians in the literature. Numerical results suggest that the dual algorithm is effective for solving nonlinear programming.

We present a nonlinear Lagrangian method for nonconvex semidefinite programming. This nonlinear Lagrangian is generated by a Lowner operator associated with Log-Sigmoid function. Under a set of assumptions, we prove a convergence theorem, which shows that the nonlinear Lagrangian algorithm is locally convergent when the penalty parameter is less than a threshold and the error bound of the solution is proportional to the penalty parameter.

We present a local convergence analysis of the method of multipliers for equality-constrained variational problems (in the special case of optimization, also called the augmented Lagrangian method) under the sole assumption that the dual starting point is close to a noncritical Lagrange multiplier (which is weaker than second-order sufficiency). Local $$Q$$ Q -superlinear convergence is established under the appropriate control of the penalty parameter values. For optimization problems, we demonstrate in addition local $$Q$$ Q -linear convergence for sufficiently large fixed penalty parameters. Both exact and inexact versions of the method are considered. Contributions with respect to previous state-of-the-art analyses for equality-constrained problems consist in the extension to the variational setting, in using the weaker noncriticality assumption instead of the usual second-order sufficient optimality condition (SOSC), and in relaxing the smoothness requirements on the problem data. In the context of optimization problems, this gives the first local convergence results for the augmented Lagrangian method under the assumptions that do not include any constraint qualifications and are weaker than the SOSC. We also show that the analysis under the noncriticality assumption cannot be extended to the case with inequality constraints, unless the strict complementarity condition is added (this, however, still gives a new result).

Targeting the adjustment of the errors-in-variables (EIV) model with equality and inequality constraints, a general solution that is similar to the classical least square adjustment is proposed based on the penalty function and the weight in measurement. Firstly, we take the equality constraints as inequality constraints that do not satisfy the constraint conditions and construct the penalty functions of equality and inequality constraints, respectively. Thus, the inequality constrained optimization problem is transformed into an unconstrained optimization problem. Then the detailed calculation formula and approximate accuracy evaluation formula of the general solution are deduced. The iteration formula of the general solution is easy regarding comprehension and applicable in implementation. It can not only solve the EIV model with equality and inequality constraints respectively, but also address the EIV model with equality and inequality constraints simultaneously. In addition, it can promote the Gauss–Markov (G-M) model with equality and inequality constraints. Finally, three examples (i.e., equality constraints, inequality constraints and those with equality and inequality constraints) are validated, indicating that the general solution is effective and feasible. The results show that the general solution is effective and feasible.

This paper establishes the relations between the changes in the values of the augmented multipliers and the changes in the values of the constraint and objective functions for a non-linear, non-convex programme based on the augmented Lagrangian which was introduced by one of the present authors. If the penalty parameters are sufficiently large, the monotonic relations hold between the augmented multipliers and the constraint functions near the optimal solution. Illustrative examples show that the classical Lagrangian fails to give rise to the relations for a. non-convex programme.

A new exact penalty method for semi-infinite programming problems

In this paper, the classical exact absolute value penalty function method is used for solving a new class of nonconvex nonsmooth optimization problems. Nonconvex nondifferentiable optimization problems with both inequality and equality constraints are considered here, in which not all functions constituting them have the fundamental property of convex functions and most classes of generalized convex functions—namely, a stationary point of such a function is its global minimum. It is proved for such nonconvex optimization problems that there exists a finite threshold of penalty parameters equal to the largest absolute value of a Lagrange multiplier such that, for every penalty parameter exceeding this lower bound, there is the equivalence between an optimal solution in the original constrained minimization problem with ‐invex functions and a minimizer in its associated penalized optimization problem with the exact l1 penalty function. Further, under nondifferentiable ‐invexity assumptions, a characterization of a saddle point of the Lagrange function, defined for the considered constrained optimization problem in terms of minimizers of its associated exact penalized optimization problem with the exact l1 penalty function, is presented.

Constrained optimization is a computationally difficult task, particularly if the constraint functions are nonlinear and non-convex. As a generic classical approach, the penalty function approach is a popular methodology which degrades the objective function value by adding a penalty proportional to the constraint violation. However, the penalty function approach has been criticized for its sensitivity to the associated penalty parameters. Since its inception, evolutionary algorithms have been modified in various ways to solve constrained optimization problems. Of them, the recent use of a bi-objective evolutionary algorithm in which the minimization of the constraint violation is included as an additional objective has received significant attention. In this article, a combination of a bi-objective evolutionary approach with the classical penalty function methodology is proposed, in a manner complementary to each other. The evolutionary approach provides an appropriate estimate of the penalty parameter, while the solution of an unconstrained penalized function by a classical method induces a convergence property to the overall hybrid algorithm. The working of the procedure on a number of standard numerical test problems and an engineering design problem is demonstrated. In most cases, the proposed hybrid methodology is observed to take one or more orders of magnitude fewer function evaluations to find the constrained minimum solution accurately than some of the best reported existing methodologies.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

When we have prior information of approximate and subjective nature other than those contained in the time series, that is knowledge not directly referable to exact linear constraints, we propose to use the penalty function method to tackle the problem of estimation of a linear statistical model alternatively to the mixed estimation proposed by Theil and Goldberger. This method bases itself on the minimization of a mathematical function known as penalty function where constraint expressed in a quadratic form is linearly combined with an objective function by some coefficients known as penalty parameters. The particular functional form utilized for the constraint can be seen as a way to catch all the uncertainty included in this type of prior knowledge. The algorithm that derives from the application of this method depends on the penalty parameters that can be interpreted in two different ways: a) as weights assigned to the constraint in the minimization procedure. In this case the result can be seen as a generalization of the ridge estimator. b) as variances of a stochastic constraint. By this way the algorithm is similar to the mixed estimator but, unlike this one, it has not unknown parameters in its formulation and so it can be seen as a feasible estimator. Whatever the interpretation of this parameters is, the penalty function method produces an algorithm that is a flexible instrument both from a theoretical viewpoint and a practical one.

Chapter 5 - Nonquadratic Penalty Functions — Convex Programming

We consider a general multi-agent convex optimization problem where the agents are to collectively minimize a global objective function subject to a global inequality constraint, a global equality constraint, and a global constraint set. The objective function is defined by a sum of local objective functions, while the global constraint set is produced by the intersection of local constraint sets. In particular, we study two cases: one where the equality constraint is absent, and the other where the local constraint sets are identical. We devise two distributed primal-dual subgradient algorithms based on the characterization of the primal-dual optimal solutions as the saddle points of the Lagrangian and penalty functions. These algorithms can be implemented over networks with dynamically changing topologies but satisfying a standard connectivity property, and allow the agents to asymptotically agree on optimal solutions and optimal values of the optimization problem under the Slater's condition.

Methodical aspects of the reliability-based structural optimisation using stochastic quasigradient methods are considered. For an example of the simply supported reinforced concrete beam, the employment of the Lagrange multiplier method that belongs to the class of stochastic quasigradient methods is demonstrated. The classical optimum design goal to minimise structural cost or weight under the constraint on the structural failure probability is taken for consideration. Optimisation problems solved with the Lagrangemultiplier method are formulated in form of general stochastic programming problem. The mathematical expectation of the concrete volume reduced with respect to the in-place cost of the beam materials is taken as the objective function. Constraint function is the limitation placed on the beam failure probability. The beam is considered as a series structural system. Values of the prescribed allowable failure probability belongs to the interval in which the estimation of the failure probabilities by the simple Monte-Carlomethod is possible with an acceptable confidence. The time-independent case as well as the time-dependent one is considered in the optimisation problems. The generalisation on the time-dependent case is undertaken through the introduction into the constraint function of the quasi-linear distribution law of the random variables. In the time-dependent case, the objective function is associated with beginning and the constraint function with end of the service period. An expression of the stochastic gradient based on the differentiation under the integral sign is used for calculations with the Lagrange multiplier method. The stochastic gradient used is computationally more effective in comparison with stochastic finite-difference formulae usual in stochastic quasigradient methods because it requires only one computation of the structure in search iteration of the optimisation process. Three rules based on statistical argumentation are used for the stopping of the seat according to the procedure of the Lagrange multiplier method. The optimising of the beam shows that the Lagrange multiplier method is applicable for the optimal design of structures in that cases when the structural reliability can be estimated by means of the simple Monte-Carlo method. Additional research is needed for integration in the Lagrange multiplier method of statistical simulation techniques for the estimation of small structural failure probabilities.