VXQR: derivative-free unconstrained optimization based on QR factorizations

Optimization solvers include many options that allow users to control algorithmic aspects that may have a considerable impact on solver performance. Tuning solver options is often necessary to reduce execution time and improve solution quality. Previous studies of solver tuning techniques have focused on mixed-integer linear programming and local nonlinear programming solvers. In this paper, we investigate the potential of tuning a global optimization solver for nonlinear and mixed-integer nonlinear programming problems. In particular, derivative-free optimization (DFO) algorithms are used to find optimal values for options of the global optimization solver BARON. A set of 126 problems from the GLOBALLib and MINLPLib collections are utilized in a computational study from which we conclude that tuning options can improve the default performance of BARON for individual problems and an entire library. Additionally, we present a systematic comparison of 27 DFO solvers in terms of their ability to improve the performance of the global solver. We find that several DFO implementations are much better than others in terms of finding optimal tuning parameters.

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 193895, “Multilevel Strategies and Geological Parameterizations for History Matching Complex Reservoir Models,” by Yimin Liu, SPE, and Louis J. Durlofsky, SPE, Stanford University, prepared for the 2019 SPE Reservoir Simulation Conference, Galveston, Texas, 10–11 April. The paper has not been peer reviewed. The complete paper explores the use of multilevel derivative-free optimization for history matching, with model properties described using principal component analysis (PCA) -based parameterization techniques. The parameterizations applied are optimization-based PCA (O-PCA) and convolutional-neural-network-based PCA (CNN-PCA). Mesh adaptive direct search (MADS), a pattern search method that parallelizes naturally, is used for the optimizations required to generate posterior models. The use of PCA-based parameterization reduces considerably the number of variables that must be determined during history matching, but the optimization problem can still be computationally demanding. The multilevel strategy addresses this issue by reducing the number of simulations that must be performed at each MADS iteration. History-matching results demonstrate that substantial uncertainty reduction is achieved in all cases considered and that the multilevel strategy is effective in reducing the number of simulations required. Overview Parameterization has two key advantages in history matching: Far fewer parameters must be determined, which simplifies greatly the minimization in some cases. Posterior (history-matched) geomodels will be consistent with the geological scenario or training image for which the parameterization was constructed. PCA is the foundation for many geological-dimension-reduction parameterization methods. PCA is based on the eigen-decomposition of the prior geomodel covariance matrix and, thus, honors only two-point spatial statistics. History-matching algorithms largely can be classified as either optimization-based or ensemble-based procedures. Optimization-based algorithms generate one history-matched model at a time, while ensemble-based algorithms update a large set of models simultaneously. The resulting ensemble of posterior models is used to quantify uncertainty. In this study, only optimization-based history matching is considered, though the parameterizations applied also can be used with ensemble-based methods. Optimization-based history-matching algorithms can be classified further into gradient-based and derivative-free optimization (DFO) methods. Gradient-based algorithms often are combined with adjoint procedures that provide the required gradient efficiently. Unlike adjoint-based procedures, DFO algorithms (which are considered in this work) are nonintrusive. They treat the reservoir simulator as a so-called “black box,” and, as a result, they are much easier to implement and can be applied with any simulator. In this study, MADS, a pattern-search DFO, is applied for history matching. In pattern-search (or stencil-based) algorithms, the number of simulations required at each iteration scales with the number of parameters to be determined. Because the latter typically is much smaller than the number of grid-blocks in the model, geological parameterization leads to large computational savings when a pattern search, or other DFO, is used for history matching. The performance of DFO for history matching with non-Gaussian models, a primary topic in the complete paper, has not been studied extensively. The authors devise a multilevel strategy, applicable for PCA-based parameterization methods, to further reduce the computational cost of history matching using DFO. A multilevel strategy lends itself naturally to PCA-based parameterizations because the PCA representation can be ordered from the most-important to the least-important principal component. Thus, PCA parameters can be determined sequentially, level by level, in groups, rather than all at once.

Direct-search algorithms form one of the main classes of algorithms for smooth unconstrained derivative-free optimization, due to their simplicity and their well-established convergence results. They proceed by iteratively looking for improvement along some vectors or directions. In the presence of smoothness, first-order global convergence comes from the ability of the vectors to approximate the steepest descent direction, which can be quantified by a first-order criticality (cosine) measure. The use of a set of vectors with a positive cosine measure together with the imposition of a sufficient decrease condition to accept new iterates leads to a convergence result as well as a worst-case complexity bound. In this paper, we present a second-order study of a general class of direct-search methods. We start by proving a weak second-order convergence result related to a criticality measure defined along the directions used throughout the iterations. Extensions of this result to obtain a true second-order optimality one are discussed, one possibility being a method using approximate Hessian eigenvectors as directions (which is proved to be truly second-order globally convergent). Numerically guaranteeing such a convergence can be rather expensive to ensure, as it is indicated by the worst-case complexity analysis provided in this paper, but turns out to be appropriate for some pathological examples.

A nonmonotone conic trust region method based on line search for solving unconstrained optimization

Black box operation optimization of basic oxygen furnace steelmaking process with derivative free optimization algorithm

All unconstrained and many constrained optimization problems involve line searches, i.e. minimizing the value of a certain function along a properly chosen direction. There are several methods for performing such onedimensional optimization but all of them require that the function be unimodal along the search interval. That may force small step sizes and in any case convergence to the closest local optimum. For multimodal functions a line search along any direction is likely to have multiple valleys. We propose using a Genetic Line Search with scalar-coded individuals, convex linear combination crossover and niche formation. Computational experiences show that this approach is more robust with respect to the starting point and that a fewer number of line searches is usually required.KeywordsSearch DirectionLine SearchUnconstrained OptimizationMultimodal FunctionSearch IntervalThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper considers the efficient minimization of the infinite time average of a stationary ergodic process in the space of a handful of design parameters which affect it. Problems of this class, derived from physical or numerical experiments which are sometimes expensive to perform, are ubiquitous in engineering applications. In such problems, any given function evaluation, determined with finite sampling, is associated with a quantifiable amount of uncertainty, which may be reduced via additional sampling. The present paper proposes a new optimization algorithm to adjust the amount of sampling associated with each function evaluation, making function evaluations more accurate (and, thus, more expensive), as required, as convergence is approached. The work builds on our algorithm for Delaunay-based Derivative-free Optimization via Global Surrogates ($${\varDelta }$$-DOGS, see JOGO https://doi.org/10.1007/s10898-015-0384-2). The new algorithm, dubbed $$\alpha $$-DOGS, substantially reduces the overall cost of the optimization process for problems of this important class. Further, under certain well-defined conditions, rigorous proof of convergence to the global minimum of the problem considered is established.

In this work, we propose Model-and-Search (MAS), a novel local-search derivative-free optimization algorithm, and show that it is convergent to a Karush-Kuhn-Tucker point. MAS aims to optimize a deterministic function over a box-bounded domain and is designed to work well within a confined budget of function evaluations. In MAS, the search is oriented to improve the value of the incumbent by combining a set of techniques, including gradient estimation and quadratic model building and optimization. We propose a novel sensitivity-based approach to construct an incomplete quadratic model when points are not enough to build a complete quadratic surrogate model of the true function. The surrogate model is then used to guide the search. We present extensive computational results on a collection of 501 publicly available test problems with varying dimensions and complexity. The computational results demonstrate that MAS performs well regardless of problem convexity and smoothness.

A robust multi-fidelity optimization methodology has been developed, focusing on efficiently handling industrial runner design of hydraulic Francis turbines. The computational task is split between low- and high-fidelity phases in order to properly balance the CFD cost and required accuracy in different design stages. In the low-fidelity phase, a physics-based surrogate optimization loop manages a large number of iterative optimization evaluations. Two derivative-free optimization methods use an inviscid flow solver as a physics-based surrogate to obtain the main characteristics of a good design in a relatively fast iterative process. The case study of a runner design for a low-head Francis turbine indicates advantages of integrating two derivative-free optimization algorithms with different local- and global search capabilities.

Many different procedures have been proposed for optimization calculations when first derivatives are not available. Further, several researchers have contributed to the subject, including some who wish to prove convergence theorems, and some who wish to make any reduction in the least calculated value of the objective function. There is not even a key idea that can be used as a foundation of a review, except for the problem itself, which is the adjustment of variables so that a function becomes least, where each value of the function is returned by a subroutine for each trial vector of variables. Therefore the paper is a collection of essays on particular strategies and algorithms, in order to consider the advantages, limitations and theory of several techniques. The subjects addressed are line search methods, the restriction of vectors of variables to discrete grids, the use of geometric simplices, conjugate direction procedures, trust region algorithms that form linear or quadratic approximations to the objective function, and simulated annealing. We study the main features of the methods themselves, instead of providing a catalogue of references to published work, because an understanding of these features may be very helpful to future research.

A modified Polak–Ribière–Polyak conjugate gradient algorithm for nonsmooth convex programs

This paper presents a numerical comparison between algorithms for unconstrained optimization that take account of sparsity in the second derivative matrix of the objective function.Some of the methods included in the comparison use difference approximation schemes to evaluate the second derivative matrix and others use an approximation to it which is updated regularly using the changes in the gradient.These results show what method to use in what circumstances and also suggest interesting future developments.1. Introduction.Numerical procedures for finding the minimum of a general function of several variables have been studied extensively in the last few years; and many powerful algorithms have been proposed, which require the calculation of the vector of first derivatives (see Davidon [2], Fletcher and Powell [4], Huang [6],Powell [8], for example).One common feature of these procedures is the fact that they use a matrix which can be regarded as an approximation to the second derivative matrix of the objective function.This matrix is updated regularly, so that the updated matrix satisfies a linear equation which is known usually as the "quasi-Newton equation" or "DFP condition".Because all the classical updating formulae revise all the elements of the matrix, the size of the problem that can be treated is often limited by the amount of computer storage that is available.However, a way to take any present sparsity into account has been suggested by Powell [11].It does not change the elements of the approximating matrix corresponding to known zero entries in the hessian of the objective function.Toint [14] shows that the calculation can be arranged so that its main part is only the solution of a sparse, symmetric and positive definite system of linear equations.The main purpose of this paper is to present numerical results that compare this procedure with some other methods that also use first derivative information, in order to show the actual and computational value of the proposed method.The other methods included in the comparison are Powell's method [8], since the new formula may be viewed as an extension of it to the sparse case, and methods using difference approximations to the hessian.When sparsity conditions

We classify Straeter's ideas for parallel unconstrained optimization and apply them to Huang's class of updating formulas. Straeter's rank-one updating formula appears to be the only parallel extension within Huang's class with the property of quadratic termination. We also develop a parallel extension of Broyden's (1965) rank-one updating formula and prove quadratic termination. Finally, we present numerical results, obtained by testing the algorithms on several standard example problems.

In Andrei [Scaled memoryless BFGS preconditioned conjugate gradient (CG) algorithm for unconstrained optimization, Optim. Meth. Softw. 22(4) (2007), pp. 561–571], an efficient CG algorithm has been proposed for solving unconstrained optimization problems. However, due to a wrong inequality used in Andrei to show the sufficient descent property for the search directions, the proof of Theorem 2, the global convergence theorem, is incorrect. In what follows, the necessary corrections will be mentioned. Throughout, we use the same notations as in Andrei.

Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization