Penalty and Augmented Lagrangian Methods

This paper presents a new point-to-face contact algorithm for contacts between two polyhedrons with planar boundaries. A new discrete numerical method called three-dimensional discontinuous deformation analysis (3-D DDA) is used and formulations of normal contact submatrices based on the proposed algorithm are derived. The presented algorithm is a simple and efficient method and it can be easily coded into a computer program. This approach does not need to use an iterative algorithm in each time step to obtain the contact plane, unlike the ‘Common-Plane’ method applied in the existing 3-D DDA. In the present 3-D DDA method, block contact constraints are enforced using the penalty method. This approach is quite simple, but may lead to inaccuracies that may be large for small values of the penalty number. The penalty method also creates block contact overlap, which violates the physical constraints of the problem. These limitations are overcome by using the augmented Lagrangian method that is used for normal contacts in this research. This point-to-face contact model has been programmed and some illustrative examples are provided to demonstrate the new contact rule between two blocks. A comparison between results obtained by using the augmented Lagrangian method and the penalty method is presented as well.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

<p style='text-indent:20px;'>Total generalization variation (TGV) is a very powerful and important regularization for various inverse problems and computer vision tasks. In this paper, we propose a semismooth Newton based augmented Lagrangian method for solving this problem. The augmented Lagrangian method (also called as method of multipliers) is widely used for lots of smooth or nonsmooth variational problems. However, its efficiency heavily depends on solving the corresponding coupled and nonlinear system together and simultaneously. With efficient primal-dual semismooth Newton methods for the challenging and highly coupled nonlinear subproblems involving total generalized variation, we develop a highly efficient and competitive augmented Lagrangian method compared with some fast first-order method. With the analysis of the metric subregularities of the corresponding functions, we give both the global convergence and local linear convergence rate for the proposed augmented Lagrangian methods.</p>

An Uzawa-type augmented Lagrangian numerical manifold method for frictional discontinuities in rock masses

This paper describes several methods for solving nonlinear complementarity problems. A general duality framework for pairs of monotone operators is developed and then applied to the monotone complementarity problem, obtaining primal, dual, and primal-dual formulations. We derive Bregman-function-based generalized proximal algorithms for each of these formulations, generating three classes of complementarity algorithms. The primal class is well-known. The dual class is new and constitutes a general collection of methods of multipliers, or augmented Lagrangian methods, for complementarity problems. In a special case, it corresponds to a class of variational inequality algorithms proposed by Gabay. By appropriate choice of Bregman function, the augmented Lagrangian subproblem in these methods can be made continuously differentiable. The primal-dual class of methods is entirely new and combines the best theoretical features of the primal and dual methods. Some preliminary computation shows that this class of algorithms is effective at solving many of the standard complementarity test problems.

Chapter 5 - Nonquadratic Penalty Functions — Convex Programming

The analysis of many engineering problems involves not only deformation of the considered system, but occurrence of the interaction between the individual separate elements of the system as well. The occurrence of friction is the most common phenomenon occurring during this interaction. In the case of intense friction, the heat released in this process is also important. In computer methods of mechanics, the process of interaction between bodies is carried out using special algorithms. The most frequently applied are: the penalty method, the barrier method, direct elimination of constraints, the Lagrange multiplier method, the perturbed Lagrangian method, the augmented Lagrangian method, Nitsche method. Owing to its easy implementation process, an approach based on a penalty function is often applied. In this approach, the contact between the bodies can be identified with the presence of the spring between the elements of the bodies in the contact. The stiffness of the spring depends on: material bulk modulus, face area, volume or shell diagonal and a numerically selected scale factor. The article will present the results of analyses that will allow fast and easy selection of its value. In the analyses there were presented the results considering the basic types of contacts: node to surface, surface to surface and surface to surface mortar.

In this paper, we propose an image segmentation model where an $L^1$ variant of the Euler's elastica energy is used as boundary regularization. An interesting feature of this model lies in its preference for convex segmentation contours. However, due to the high order and non-differentiability of Euler's elastica energy, it is nontrivial to minimize the associated functional. As in recent work on the ordinary $L^2$-Euler's elastica model in imaging, we propose using an augmented Lagrangian method to tackle the minimization problem. Specifically, we design a novel augmented Lagrangian functional that deals with the mean curvature term differently than in previous works. The new treatment reduces the number of Lagrange multipliers employed, and more importantly, it helps represent the curvature more effectively and faithfully. Numerical experiments validate the efficiency of the proposed augmented Lagrangian method and also demonstrate new features of this particular segmentation model, such as shape driven and data driven properties.

Optimal investment strategies for enhanced indexation problems have attracted considerable attentions over the last decades in the field of fund management. In this paper, a featured difference from the existing literature is that our main concern of the investigation is the development of a sparse-enhanced indexation model to describe the process of assets selection by introducing a sparse regularization instead of binary variables, which is expected to avoid the over-fitting and promote a better out-of-sample performance for the resulting tracking portfolio to some extent. An Alternating Quadratic Penalty (AQP) method is proposed to solve the corresponding nonconvex optimisation problem, into which the Block Coordinate Descent (BCD) algorithm is integrated to solve a sequence of penalty subproblems. Under some suitable assumptions, we establish that any accumulation point of the sequence generated by the AQP method is a KKT point of the proposed model. Computational results on five typical data-sets are reported to verify the efficiency of the proposed AQP method, including the superiority of the sparse model with the AQP method over one cardinality constrained quadratic programming model with one of its solution methods in terms of computational costs, out-of-sample performances, and the consistency between in-sample and out-of-sample performances of the resulting tracking portfolios.

In this paper, a reduced order modeling framework for nonlinear vibration problems of elastic structures involving intermittent contact is proposed. Of particular interest is a vibration problem of plate-like elastic structures with a crack with a large number of degrees of freedom involved on the crack surfaces. Due to the localized nature of such nonlinearity, the number of degrees of freedom on the surfaces greatly affects the computational time of the analysis. Therefore, reducing the number of degrees of freedom on the crack surfaces without significantly sacrificing the accuracy of the results is a critical issue for conducting vibration analysis of such structures in a reasonable amount of time. The focus is placed on the development of an efficient algorithm to select a set of nodes on the crack surfaces, where nonlinear boundary conditions are imposed. The method is developed based on a procedure for selecting master degrees of freedom for Guyan reduction. The accuracy, efficiency, and optimality of the method are discussed in detail and compared with those aspects of previous methods. The advantages of the new method are demonstrated in terms of the accuracy of the frequency response and the resonant frequencies. IBRATION problems of structures with intermittent contact have been studied extensively for several decades. These problems have practical importance and feature theoretical complexity due to their nonlinear nature. A numerical modeling procedure of such problems based on the finite element (FE) method is presented in this paper. This work is motivated by a need for developing a model-based crack detection algorithm of elastic structures based on their spectral properties, such as resonant frequencies and response shapes. In order to properly predict the resonant frequencies of such structures, one has to consider the nonlinearity caused by intermittent contact at the cracks, the so-called closing crack or breathing crack effect. This has hindered analysts from accurately calculating the resonant frequencies of cracked structures, because they cannot be calculated from classical linear modal analysis. As some sophisticated contact algorithms have been developed, such as the penalty method 1 and the augmented Lagrangian method, 2 the accuracy of the results of time transient simulation with FE models involving intermittent contact has been improved. Furthermore, studying vibration problems of such structures with an FE model with a realistic complexity is becoming feasible with the aid of high-performance computers. However, in turn, due to

Objective energy–momentum conserving integration for the constrained dynamics of geometrically exact beams

In this paper a fully explicit finite element method (FEFEM) is presented for solving steady incompressible viscous flow problems. This full explicitness is achieved by combining the multiplier (or augmented Lagrangian) method with a pseudo‐time‐iteration method. FEFEM needs no global matrix at all and is of great advantage to large‐scale problems because they can be solved within the limit of core memory.The optimum choice of a time increment and a penalty parameter is discussed and the driven cavity flow at a Reynolds number of 1000 is computed with a refined mesh (60 × 60 elements).

Due to the possible lack of primal-dual-type error bounds, it was not clear whether the Karush–Kuhn–Tucker (KKT) residuals of the sequence generated by the augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) problems converge superlinearly. In this paper, we resolve this issue by establishing the R-superlinear convergence of the KKT residuals generated by the ALM under only a mild quadratic growth condition on the dual of CCCP, with easy-to-implement stopping criteria for the augmented Lagrangian subproblems. This discovery may help to explain the good numerical performance of several recently developed semismooth Newton-CG based ALM solvers for linear and convex quadratic semidefinite programming.

Because balanced constraints can overcome the problems of trivial solutions of data classification via minimum cut method, many techniques with different balanced strategies have been proposed to improve data classification accuracy. However, their performances have not been compared comprehensively so far. In this paper, we investigate seven balanced classification methods under the discrete non-local total variational framework and compare their accuracy performances on graph. The two-class classification problem with equality constraints, inequality constraints and Ratio Cut, Normalized Cut, Cheeger Cut models are investigated. For cases of equality constraint, we firstly compare the Penalty Function Method (PFM) and the Augmented Lagrangian Method (ALM), which can transform the constrained problems into unconstrained ones, to show the advantages of ALM. The other cases are all solved using the ALM also. In order to make the comparison fairly, we solve all models using ALM method and using the same proportion of fidelity points and the same neighborhood size on graph. Experimental results demonstrate ALM with the equality balanced constraint has the best classification accuracy compared with other six constraints. 200 words.

We present a stochastic approximation algorithm based on penalty function method and a simultaneous perturbation gradient estimate for solving stochastic optimisation problems with general inequality constraints. We present a general convergence result that applies to a class of penalty functions including the quadratic penalty function, the augmented Lagrangian, and the absolute penalty function. We also establish an asymptotic normality result for the algorithm with smooth penalty functions under minor assumptions. Numerical results are given to compare the performance of the proposed algorithm with different penalty functions.