A hybrid differential evolution augmented Lagrangian method for constrained numerical and engineering optimization

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Augmented Lagrangian methods are effective tools for solving large-scale nonlinear programming problems. At each outer iteration, a minimization subproblem with simple constraints, whose objective function depends on updated Lagrange multipliers and penalty parameters, is approximately solved. When the penalty parameter becomes very large, solving the subproblem becomes difficult; therefore, the effectiveness of this approach is associated with the boundedness of the penalty parameters. In this paper, it is proved that under more natural assumptions than the ones employed until now, penalty parameters are bounded. For proving the new boundedness result, the original algorithm has been slightly modified. Numerical consequences of the modifications are discussed and computational experiments are presented.

We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hestenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality constraints, but it can use the second derivatives and does not depend on projections on the set of inequality constraints. In each iteration, our method solves a twice continuously differentiable unconstrained optimization subproblem on primal variables. The dual iterates, penalty and smoothing parameters are updated adaptively. The global and local convergence are analyzed. Without assuming any constraint qualification, it is proved that the proposed method has strong global convergence. The method may converge to either a Karush-Kuhn-Tucker (KKT) point or a singular stationary point when the converging point is a minimizer. It may also converge to an infeasible stationary point of nonlinear program when the problem is infeasible. Furthermore, our method is capable of rapidly detecting the possible infeasibility of the solved problem. Under suitable conditions, it is locally linearly convergent to the KKT point, which is consistent with ALMs for optimization with equality constraints. The preliminary numerical experiments on some small benchmark test problems demonstrate our theoretical results.

The differential evolution (DE) is an improved version of evolution strategies and Nelder-Mead simplex methods. DE has been successfully applied in various fields, such as optimization nonlinear functions, multi sensor fusion, control system, and system identification. The potentialities of DE are its simple structure, easy use, convergency speed and robustness. This paper proposes a new hybrid methodology for solving the economic load dispatch problem with valvepoint effect. The proposed hybrid method combinates the DE and sequential quadratic programming (SQP) technique. The DE is used to produce good potential solutions, and the SQP is used to fine-tune the DE run. The hybrid methodology and its variants are validated for a test system consisting of 40 thermal units whose incremental fuel cost function takes into account the valvepoint loading effects. The proposed hybrid method outperforms other state-of-theart algorithms in solving load dispatch problems with the valve-point effect.Keywordsdifferential evolutioneconomic dispatchoptimizationpower generation operation and control

ABSTRACTThe proximal method of multipliers was proposed by Rockafellar [Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math Oper Res. 1976;1:97–116] for solving convex programming and it is a kind of proximal point method applied to convex programming. In this paper, we apply this method for solving nonlinear semidefinite programming problems, in which subproblems have better properties than those from the augmented Lagrange method. We prove that, under the linear independence constraint qualification and the strong second-order sufficiency optimality condition, the rate of convergence of the proximal method of multipliers, for a nonlinear semidefinite programming problem, is linear and the ratio constant is proportional to 1/c, where c is the penalty parameter that exceeds a threshold . Moreover, the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to .

A deterministic global optimization algorithm based on augmented Lagrangian is proposed for solving the nonlinear programming with cone constraints, which is often encountered in technical design and operational research. At each outer iteration, the method requires the -global minimization of the augmented Lagrangian of problems with cone constraints, where. The convergence to an-global minimizer of the original problem is proved. Key words: Global optimization, augmented Lagrangians, cone optimization, algorithms.

In this work, a hybrid method that includes metaheuristic algorithms has been proposed for solving optimization problems. The proposed method was implemented as employing three metaheuristic algorithms which are Artificial Bee Colony, Differential Evolution and Particle Swarm Optimization in an order. The success of the developed method is presented by testing on 12 continuous optimization test functions which are widely used in the literature. The experimental results show that the proposed hybrid method gives better results than the individual algorithms that make up it.

In this paper, we propose an image segmentation model where an $L^1$ variant of the Euler's elastica energy is used as boundary regularization. An interesting feature of this model lies in its preference for convex segmentation contours. However, due to the high order and non-differentiability of Euler's elastica energy, it is nontrivial to minimize the associated functional. As in recent work on the ordinary $L^2$-Euler's elastica model in imaging, we propose using an augmented Lagrangian method to tackle the minimization problem. Specifically, we design a novel augmented Lagrangian functional that deals with the mean curvature term differently than in previous works. The new treatment reduces the number of Lagrange multipliers employed, and more importantly, it helps represent the curvature more effectively and faithfully. Numerical experiments validate the efficiency of the proposed augmented Lagrangian method and also demonstrate new features of this particular segmentation model, such as shape driven and data driven properties.

Constrained optimization problems are very important as they are encountered in many science and engineering applications. A hybrid method based on modified augmented Lagran- gian multiplier and biogeography-based optimization (BBO) algorithm is proposed to solve con- strained optimization problems. The basic steps of the proposed method are comprised of an outer iteration, in which the Lagrangian multipliers and various penalty parameters are updated using a first-order update scheme, and an inner iteration, in which a nonlinear optimization of the modified augmented Lagrangian function with simple bound constraints is implemented by BBO algorithm. Numerical results show that the proposed method is reliable and efficient for solving constrained optimization problems.

Given the ultrahigh dimensionality and the complex structure, which contains matrices and vectors, the mixed matrix minimization becomes crucial for the analysis of those data. Recently, the nonconvex functions such as the smoothly clipped absolute deviation, the minimax concave penalty, the capped $ \ell_1 $-norm penalty and the $ \ell_p $ quasi-norm with $ 0<p<1 $ have been shown remarkable advantages in variable selection due to the fact that they can overcome the over-penalization. In this paper, we propose and study a novel nonconvex mixed matrix minimization, which combines the low-rank and sparse regularzations and nonconvex functions perfectly. The augmented Lagrangian method (ALM) is proposed to solve the noncovnex mixed matrix minimization problem. The resulting subproblems either have closed-form solutions or can be solved by fast solvers, which makes the ALM particularly efficient. In theory, we prove that the sequence generated by the ALM converges to a stationary point when the penalty parameter is above a computable threshold. Extensive numerical experiments illustrate that our proposed nonconvex mixed matrix minimization model outperforms the existing ones.

In this paper a fully explicit finite element method (FEFEM) is presented for solving steady incompressible viscous flow problems. This full explicitness is achieved by combining the multiplier (or augmented Lagrangian) method with a pseudo‐time‐iteration method. FEFEM needs no global matrix at all and is of great advantage to large‐scale problems because they can be solved within the limit of core memory.The optimum choice of a time increment and a penalty parameter is discussed and the driven cavity flow at a Reynolds number of 1000 is computed with a refined mesh (60 × 60 elements).

In this paper, we propose a distributed algorithm for optimization problems that involve a separable, possibly nonconvex objective function subject to convex local constraints and linear coupling constraints. The method is based on the accelerated distributed augmented Lagrangians (ADAL) algorithm that was recently developed by the authors to address convex problems. Here, we extend this line of work in two ways. First, we establish convergence of the method to a local minimum of the problem, using assumptions that are common in the analysis of nonconvex optimization methods. To the best of our knowledge, this is the first work that shows convergence to local minima specifically for a distributed augmented Lagrangian (AL) method applied to nonconvex optimization problems; distributed AL methods are known to perform very well when used to solve convex problems. Second, we propose a more general and decentralized rule to select the stepsizes of the method. This improves on the authors’ original ADAL method, where the stepsize selection used global information at initialization. Numerical results are included to verify the correctness and efficiency of the proposed distributed method.

Due to the possible lack of primal-dual-type error bounds, it was not clear whether the Karush–Kuhn–Tucker (KKT) residuals of the sequence generated by the augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) problems converge superlinearly. In this paper, we resolve this issue by establishing the R-superlinear convergence of the KKT residuals generated by the ALM under only a mild quadratic growth condition on the dual of CCCP, with easy-to-implement stopping criteria for the augmented Lagrangian subproblems. This discovery may help to explain the good numerical performance of several recently developed semismooth Newton-CG based ALM solvers for linear and convex quadratic semidefinite programming.

We present a local convergence analysis of the method of multipliers for equality-constrained variational problems (in the special case of optimization, also called the augmented Lagrangian method) under the sole assumption that the dual starting point is close to a noncritical Lagrange multiplier (which is weaker than second-order sufficiency). Local $$Q$$ Q -superlinear convergence is established under the appropriate control of the penalty parameter values. For optimization problems, we demonstrate in addition local $$Q$$ Q -linear convergence for sufficiently large fixed penalty parameters. Both exact and inexact versions of the method are considered. Contributions with respect to previous state-of-the-art analyses for equality-constrained problems consist in the extension to the variational setting, in using the weaker noncriticality assumption instead of the usual second-order sufficient optimality condition (SOSC), and in relaxing the smoothness requirements on the problem data. In the context of optimization problems, this gives the first local convergence results for the augmented Lagrangian method under the assumptions that do not include any constraint qualifications and are weaker than the SOSC. We also show that the analysis under the noncriticality assumption cannot be extended to the case with inequality constraints, unless the strict complementarity condition is added (this, however, still gives a new result).

<p style='text-indent:20px;'>Total generalization variation (TGV) is a very powerful and important regularization for various inverse problems and computer vision tasks. In this paper, we propose a semismooth Newton based augmented Lagrangian method for solving this problem. The augmented Lagrangian method (also called as method of multipliers) is widely used for lots of smooth or nonsmooth variational problems. However, its efficiency heavily depends on solving the corresponding coupled and nonlinear system together and simultaneously. With efficient primal-dual semismooth Newton methods for the challenging and highly coupled nonlinear subproblems involving total generalized variation, we develop a highly efficient and competitive augmented Lagrangian method compared with some fast first-order method. With the analysis of the metric subregularities of the corresponding functions, we give both the global convergence and local linear convergence rate for the proposed augmented Lagrangian methods.</p>