Algorithms for Constrained and Weighted Nonlinear Least Squares

Preface 1. Introduction. Problems to be considered Characteristics of 'real-world' problems Finite-precision arithmetic and measurement of error Exercises 2. Nonlinear Problems in One Variable. What is not possible Newton's method for solving one equation in one unknown Convergence of sequences of real numbers Convergence of Newton's method Globally convergent methods for solving one equation in one uknown Methods when derivatives are unavailable Minimization of a function of one variable Exercises 3. Numerical Linear Algebra Background. Vector and matrix norms and orthogonality Solving systems of linear equations-matrix factorizations Errors in solving linear systems Updating matrix factorizations Eigenvalues and positive definiteness Linear least squares Exercises 4. Multivariable Calculus Background Derivatives and multivariable models Multivariable finite-difference derivatives Necessary and sufficient conditions for unconstrained minimization Exercises 5. Newton's Method for Nonlinear Equations and Unconstrained Minimization. Newton's method for systems of nonlinear equations Local convergence of Newton's method The Kantorovich and contractive mapping theorems Finite-difference derivative methods for systems of nonlinear equations Newton's method for unconstrained minimization Finite difference derivative methods for unconstrained minimization Exercises 6. Globally Convergent Modifications of Newton's Method. The quasi-Newton framework Descent directions Line searches The model-trust region approach Global methods for systems of nonlinear equations Exercises 7. Stopping, Scaling, and Testing. Scaling Stopping criteria Testing Exercises 8. Secant Methods for Systems of Nonlinear Equations. Broyden's method Local convergence analysis of Broyden's method Implementation of quasi-Newton algorithms using Broyden's update Other secant updates for nonlinear equations Exercises 9. Secant Methods for Unconstrained Minimization. The symmetric secant update of Powell Symmetric positive definite secant updates Local convergence of positive definite secant methods Implementation of quasi-Newton algorithms using the positive definite secant update Another convergence result for the positive definite secant method Other secant updates for unconstrained minimization Exercises 10. Nonlinear Least Squares. The nonlinear least-squares problem Gauss-Newton-type methods Full Newton-type methods Other considerations in solving nonlinear least-squares problems Exercises 11. Methods for Problems with Special Structure. The sparse finite-difference Newton method Sparse secant methods Deriving least-change secant updates Analyzing least-change secant methods Exercises Appendix A. A Modular System of Algorithms for Unconstrained Minimization and Nonlinear Equations (by Robert Schnabel) Appendix B. Test Problems (by Robert Schnabel) References Author Index Subject Index.

Preface. Solving Complementarity Problems by Means of a New Smooth Constrained Nonlinear Solver R. Andreani, J.M. Martinez.epsiv -Enlargements of Maximal Monotone Operators: Theory and Applications R.S. Burachik, et al. A Non-Interior Predictor Corrector Path-Following Method for LCP J.V. Burke, S. Xu.Smoothing Newton Methods for Nonsmooth Dirichlet Problems X. Chen, etal. Frictional Contact Algorithms Based on Semismooth Newton Methods P.W. Christensen, J.-S. Pang. Well-Posed Problems and Error Bounds in Optimization S. Deng. Modeling and Solution Environments for MPEC: GAMS & MATLAB S.P. Dirkse, M.C. Ferris. Merit Functions and Stability for Complementarity Problems A. Fischer. Minimax and Triality Theory in Nonsmooth Variational Problems D.Y. Gao. Global and Local Superlinear Convergence Analysis of Newton-Type Methods for Semismooth Equations with Smooth Least Squares H. Jiang, D. Ralph.Inexact Trust-Region Methods for Nonlinear Complementarity Problems C.Kanzow, M. Zupke. Regularized Newton Methods for Minimization of Convex Quadratic Splines with Singular Hessians W. Li, J. Swetits.Regularized Linear Programs with Equilibrium Constraints L.L.Mangasarian. Reformulations of a Bicriterion Equilibrium Model P.Marcotte. A Smoothing Function and Its Applications J.-M. Peng.On the Local Super-Linear Convergence of a Matrix Secant Implementation of the Variable Metric Proximal Point Algorithm for Monotone Operators M.Qian, J.V. Burke. Reformulation of a Problem of Economic Equilibrium A.M. Rubinov, B.M. Glover. A Globally Convergent Inexact NewtonMethod for Systems of Monotone Equations M.V. Solodov, B.F.Svaiter. On the Limiting Behavior of the Trajectory of RegularizedSolutions of a P0 -Complementarity Problem R. Sznajder, M.S.Gowda. Analysis of a Non-Interior Continuation Method Based on Chen-Mangasarian Smoothing Functions for Complementarity Problems P.Tseng. A New Merit Function and a Descent Method for Semidefinite Complementarity Problems N. Yamashita, M. Fukushima. Numerical Experiments for a Class of Squared Smoothing Newton Methods for Box Constrained Variational Inequality Problems G. Zhou, et al.

Unifying semilocal and local convergence of Newton's method on Banach space with a convergence structure

Newton’s method is frequently employed as a means to solve nonlinear equations because of its rapid local convergence properties; however, the behaviour of Newton's method is very much dependent upon how close the initial guess is to the true solution. An inappropriate or poor initial guess can severely hinder the convergence of Newton’s method. A novel hybrid numerical/data driven approach to estimating the initial guess that can be used in conjunction with Newton's method is proposed, whereby a machine learning model is trained to make predictions about the location of the root of a specified function from sampled function values taken from a bounded interval. To develop this approach, a synthetic dataset of nonlinear equations was produced and the ML algorithm was trained to predict the location of the root, based on these training examples, that would be subsequently used as the initial guess for the numerical solver. Experimental results indicate that the proposed hybrid numerical/data-driven approach will increase the success rate of obtaining a solution with Newton's method from roughly 97% to approximately 99.75% and decrease the average number of iterations required to obtain a solution by over 30% when compared to standard methods of initialization (midpoint or random guesses). Thus, the results presented in this paper indicate that data driven initial guessing can improve both the robustness and the efficiency of classical numerical methods, without changing the classical mathematical constructs by which they operate.

SUMMARY In many problems of maximum likelihood estimation, it is impossible to carry out either the E-step or the M-step of the EM algorithm. The present paper introduces a gradient algorithm that is closely related to the EM algorithm. This EM gradient algorithm approximately solves the M-step of the EM algorithm by one iteration of Newton's method. Since Newton's method converges quickly, the local properties of the EM gradient algorithm are almost identical with those of the EM algorithm. Any strict local maximum point of the observed likelihood locally attracts the EM and EM gradient algorithm at the same rate of convergence, and near the maximum point the EM gradient algorithm always produces an increase in the likelihood. With proper modification the EM gradient algorithm also exhibits global convergence properties that are similar to those of the EM algorithm. Our proof of global convergence applies and improves existing theory for the EM algorithm. These theoretical points are reinforced by a discussion of three realistic examples illustrating how the EM gradient algorithm can succeed where the EM algorithm is intractable.

Exact Newton method with third-order convergence to model the dynamics of bubbles in incompressible flow

Numerical solution of nonlinear least-squares problems is an important computational task in science and engineering. Effective algorithms have been developed for solving nonlinear least squares problems. The structured secant method is a class of efficient methods developed in recent years for optimization problems in which the Hessian of the objective function has some special structure. A primary and typical application of the structured secant method is to solve the nonlinear least squares problems. We present an exact penalty method for solving constrained nonlinear leastsquares problems, when the structured projected Hessian is approximated by a projected version of the structured BFGS formula and give its local two-step Q-superlinear convergence. For robustness, we employ a special nonsmooth line search strategy, taking account of the least squares objective. We discuss the comparative results of the testing of our programs and three nonlinear programming codes from KNITRO on some randomly generated test problems due to Bartels and MahdaviAmiri. Numerical results also confirm the practical relevance of our special considerations for the inherent structure of the least squares.

Location, separation and approximation of solutions of nonlinear Hammerstein-type integral equations

While there already exist randomized subspace Newton methods that restrict the search direction to a random subspace for a convex function, we propose a randomized subspace regularized Newton method for a non-convex function and more generally we investigate thoroughly, for the first time, the local convergence rate of the randomized subspace Newton method. In our proposed algorithm, we use a modified Hessian of the function restricted to some random subspace so that, with high probability, the function value decreases at each iteration, even when the objective function is non-convex. We show that our method has global convergence under appropriate assumptions and its convergence rate is the same as that of the full regularized Newton method. Furthermore, we obtain a local linear convergence rate under some additional assumptions, and prove that this rate is the best we can hope, in general, when using a random subspace. We furthermore prove that if the Hessian, at the local optimum, is rank deficient then super-linear convergence holds.

An equation-by-equation (EBE) method is proposed to solve a system of nonlinear equations arising from the moment constrained maximum entropy problem of multidimensional variables. The design of the EBE method combines ideas from homotopy continuation and Newton's iterative methods. Theoretically, we establish the local convergence under appropriate conditions and show that the proposed method, geometrically, finds the solution by searching along the surface corresponding to one component of the nonlinear problem. We will demonstrate the robustness of the method on various numerical examples, including: (1) A six-moment one-dimensional entropy problem with an explicit solution that contains components of order $10^0-10^3$ in magnitude; (2) Four-moment multidimensional entropy problems with explicit solutions where the resulting systems to be solved ranging from $70-310$ equations; (3) Four- to eight-moment of a two-dimensional entropy problem, which solutions correspond to the densities of the two leading EOFs of the wind stress-driven large-scale oceanic model. In this case, we find that the EBE method is more accurate compared to the classical Newton's method, the MATLAB generic solver, and the previously developed BFGS-based method, which was also tested on this problem. (4) Four-moment constrained of up to five-dimensional entropy problems which solutions correspond to multidimensional densities of the components of the solutions of the Kuramoto-Sivashinsky equation. For the higher dimensional cases of this example, the EBE method is superior because it automatically selects a subset of the prescribed moment constraints from which the maximum entropy solution can be estimated within the desired tolerance. This selection feature is particularly important since the moment constrained maximum entropy problems do not necessarily have solutions in general.

Newton's method for sections on Riemannian manifolds: Generalized covariant [formula omitted]-theory

On a theorem of L.V. Kantorovich concerning Newton's method

We present local and semilocal convergence results for Newton's method in a Banach space setting. In particular, using Lipschitz-type assumptions on the second Fréchet-derivative we find results concerning the radius of convergence of Newton's method. Such results are useful in the context of predictor–corrector continuation procedures. Finally, we provide numerical examples to show that our results can apply where earlier ones using Lipschitz assumption on the first Fréchet-derivative fail.

A Newton-Lanczos method for solution of non-linear finite element equations

Communication-efficient distributed cubic Newton with compressed lazy Hessian