Model-and-search: a derivative-free local optimization algorithm

Optimization solvers include many options that allow users to control algorithmic aspects that may have a considerable impact on solver performance. Tuning solver options is often necessary to reduce execution time and improve solution quality. Previous studies of solver tuning techniques have focused on mixed-integer linear programming and local nonlinear programming solvers. In this paper, we investigate the potential of tuning a global optimization solver for nonlinear and mixed-integer nonlinear programming problems. In particular, derivative-free optimization (DFO) algorithms are used to find optimal values for options of the global optimization solver BARON. A set of 126 problems from the GLOBALLib and MINLPLib collections are utilized in a computational study from which we conclude that tuning options can improve the default performance of BARON for individual problems and an entire library. Additionally, we present a systematic comparison of 27 DFO solvers in terms of their ability to improve the performance of the global solver. We find that several DFO implementations are much better than others in terms of finding optimal tuning parameters.

A novel assisted-history-matching (AHM) approach is developed to efficiently find multiple local minima of the objective function, by performing Gauss-Newton (GN) minimizations concurrently and and sharing information from dispersed regions in parameter space dynamically. To start, a large number of different initial parameter values (i.e., model realizations) are randomly generated and are used as base-cases for each realization. Production data for all realizations are obtained by simulating these base-cases concurrently. A local quadratic model around each base-case is constructed using the GN formulation, where the required sensitivity-matrix is approximated by linear interpolation of non-degenerated points that are closest to the given base-case. New search points are generated by minimizing the local quadratic approximate models. The base-cases are updated iteratively if their corresponding search points improve the data mismatch. Finally, each base case will converge to a local minimum in the vicinity of the initial base-case. The proposed approach is applied to different test problems. Most local minima of these test problems are found with satisfactory accuracy. Compared to traditional AHM approaches using derivative-free optimization algorithms using multiple initial start values, the propose approach may achieve a reduction factor in computer resource usage that is proportional to the number of parameters.

One particular class of derivative-free optimization algorithms is trust-region algorithms based on quadratic models given by the under-determined interpolation. Different techniques in updating the quadratic model from iteration to iteration will give different interpolation models. We propose a new way to update the quadratic model by minimizing the $H^{2}$ norm of the difference between neighboring quadratic models. The motivation for applying the $H^{2}$ norm is given. The theoretical properties of our new updating technique are also presented. We propose the projection in the sense of $H^{2}$ norm and the interpolation error analysis of our model function. We obtain the coefficients of the quadratic model function using the Karush–Kuhn–Tucker (KKT) conditions. Numerical results show the advantages of our model on the test set considered, and the derivative-free algorithms based on our least $H^{2}$ norm updating quadratic model functions can solve test problems with fewer function evaluations than the algorithm based on the least Frobenius norm updating model and the other compared methods.

Black box operation optimization of basic oxygen furnace steelmaking process with derivative free optimization algorithm

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 193895, “Multilevel Strategies and Geological Parameterizations for History Matching Complex Reservoir Models,” by Yimin Liu, SPE, and Louis J. Durlofsky, SPE, Stanford University, prepared for the 2019 SPE Reservoir Simulation Conference, Galveston, Texas, 10–11 April. The paper has not been peer reviewed. The complete paper explores the use of multilevel derivative-free optimization for history matching, with model properties described using principal component analysis (PCA) -based parameterization techniques. The parameterizations applied are optimization-based PCA (O-PCA) and convolutional-neural-network-based PCA (CNN-PCA). Mesh adaptive direct search (MADS), a pattern search method that parallelizes naturally, is used for the optimizations required to generate posterior models. The use of PCA-based parameterization reduces considerably the number of variables that must be determined during history matching, but the optimization problem can still be computationally demanding. The multilevel strategy addresses this issue by reducing the number of simulations that must be performed at each MADS iteration. History-matching results demonstrate that substantial uncertainty reduction is achieved in all cases considered and that the multilevel strategy is effective in reducing the number of simulations required. Overview Parameterization has two key advantages in history matching: Far fewer parameters must be determined, which simplifies greatly the minimization in some cases. Posterior (history-matched) geomodels will be consistent with the geological scenario or training image for which the parameterization was constructed. PCA is the foundation for many geological-dimension-reduction parameterization methods. PCA is based on the eigen-decomposition of the prior geomodel covariance matrix and, thus, honors only two-point spatial statistics. History-matching algorithms largely can be classified as either optimization-based or ensemble-based procedures. Optimization-based algorithms generate one history-matched model at a time, while ensemble-based algorithms update a large set of models simultaneously. The resulting ensemble of posterior models is used to quantify uncertainty. In this study, only optimization-based history matching is considered, though the parameterizations applied also can be used with ensemble-based methods. Optimization-based history-matching algorithms can be classified further into gradient-based and derivative-free optimization (DFO) methods. Gradient-based algorithms often are combined with adjoint procedures that provide the required gradient efficiently. Unlike adjoint-based procedures, DFO algorithms (which are considered in this work) are nonintrusive. They treat the reservoir simulator as a so-called “black box,” and, as a result, they are much easier to implement and can be applied with any simulator. In this study, MADS, a pattern-search DFO, is applied for history matching. In pattern-search (or stencil-based) algorithms, the number of simulations required at each iteration scales with the number of parameters to be determined. Because the latter typically is much smaller than the number of grid-blocks in the model, geological parameterization leads to large computational savings when a pattern search, or other DFO, is used for history matching. The performance of DFO for history matching with non-Gaussian models, a primary topic in the complete paper, has not been studied extensively. The authors devise a multilevel strategy, applicable for PCA-based parameterization methods, to further reduce the computational cost of history matching using DFO. A multilevel strategy lends itself naturally to PCA-based parameterizations because the PCA representation can be ordered from the most-important to the least-important principal component. Thus, PCA parameters can be determined sequentially, level by level, in groups, rather than all at once.

We have developed a new derivative-free algorithm based on Radial Basis Functions (RBFs). Derivative-free optimization is an active field of research and several algorithms have been proposed recently. Problems of this nature in the industrial setting are quite frequent. The reason is that in a number of applications the optimization process contains simulation packages which are treated as black boxes. The development of our own algorithm was originally motivated by an application in biomedical imaging: the medical image registration problem. The particular characteristics of this problem have incited us to develop a new optimization algorithm based on trust-region methods. However it has been designed to be generic and to be applied to a wide range of problems. The main originality of our approach is the use of RBFs to build the models. In particular we have adapted the existing theory based on quadratic models to our own models and developed new procedures especially designed for models based on RBFs. We have tested our algorithm called BOOSTERS against state-of-the-art methods (UOBYQA, NEWUOA, DFO). On the medical image registration problem, BOOSTERS appears to be the method of choice. The tests on problems from the CUTEr collection show that BOOSTERS is comparable to, but not better than other methods on small problems (size 2-20). It is performing very well for medium size problems (20-80). Moreover, it is able to solve problems of dimension 200, which is considered very large in derivative-free optimization. We have also developed a new class of algorithms combining the robustness of derivative-free algorithms with the faster rate of convergence characterizing Newtonlike-methods. In fact, they define a new class of algorithms lying between derivative-free optimization and quasi-Newton methods. These algorithms are built on the skeleton of our derivative-free algorithm but they can incorporate the gradient when it is available. They can be interpreted as a way of doping derivative-free algorithms with derivatives. If the derivatives are available at each iteration, then our method can be seen as an alternative to quasi-Newton methods. At the opposite, if the derivatives are never evaluated, then the algorithm is totally similar to BOOSTERS. It is a very interesting alternative to existing methods for problems whose objective function is expensive to evaluate and when the derivatives are not available. In this situation, the gradient can be approximated by finite differences and its costs corresponds to n additional function evaluations assuming that Rn is the domain of definition of the objective function. We have compared our method with CFSQP and BTRA, two gradient-based algorithms, and the results show that our doped method performs best. We have also a theoretical analysis of the medical image registration problem based on maximization of mutual information. Most of the current research in this field is concentrated on registration based on nonlinear image transformation. However, little attention has been paid to the theoretical properties of the optimization problem. In our analysis, we focus on the continuity and the differentiability of the objective function. We show in particular that performing a registration without extension of the reference image may lead to discontinuities in the objective function. But we demonstrate that, under some mild assumptions, the function is differentiable almost everywhere. Our analysis is important from an optimization point of view and conditions the choice of a solver. The usual practice is to use generic optimization packages without worrying about the differentiability of the objective function. But the use of gradient-based methods when the objective function is not differentiable may result in poor performance or even in absence of convergence. One of our objectives with this analysis is also that practitioners become aware of these problems and to propose them new algorithms having a potential interest for their applications.

Optimization is one of the most important problems in engineering applications. Substantial algorithms have been proposed to tackle these problems, such as differential evolution(DE), which has excellent global search capability and robustness. However, like most metaheuristics, differential evolution is computationally expensive and difficult to converge to the exact global optimum. The modified Powell(MP) method is a derivative-free local optimization algorithm with fast convergence speed but is prone to fall into the local optimum trap. Combining the advantages of these two algorithms, we propose a hybrid optimization algorithm, which is called the DEMP algorithm. The DEMP algorithm is compared to other three well-known optimization algorithms by a set of benchmark functions. The results show that DEMP is not only notably better than the traditional DE Algorithm, but also highly competitive for other metaheuristic algorithms.

This paper presents basic features of a new family of algorithms for unconstrained derivative-free optimization, based on line searches along directions generated from QR factorizations of past direction matrices. Emphasis is on fast descent with a low number of function values, so that the algorithm can be used for fairly expensive functions. The theoretical total time overhead needed per function evaluation is of order O(n 2), where n is the problem dimension, but the observed overhead is much smaller. Numerical results are given for a particular algorithm VXQR1 from this family, implemented in Matlab, and evaluated on the scalability test set of Herrera et al. ( http://www.sci2s.ugr.es/eamhco/CFP.php, 2010) for problems in dimensions n ∈ {50, 100, 200, 500, 1,000}. Performance depends a lot on the graph $$\{(t,f(x+th))\mid t\in[0,1]\}$$ of the function along line segments. The algorithm is typically very fast on smooth problems with not too rugged graphs, and on problems with a roughly separable structure. It typically performs poorly on problems where the graph along many directions is highly multimodal without pronounced overall slope (e.g., for smooth functions with superimposed oscillations of significant size), where the graphs along many directions are piecewise constant (e.g., for problems minimizing a maximum norm), or where the function overflows on the major part of the search region and no starting point with finite function value is known.

A robust multi-fidelity optimization methodology has been developed, focusing on efficiently handling industrial runner design of hydraulic Francis turbines. The computational task is split between low- and high-fidelity phases in order to properly balance the CFD cost and required accuracy in different design stages. In the low-fidelity phase, a physics-based surrogate optimization loop manages a large number of iterative optimization evaluations. Two derivative-free optimization methods use an inviscid flow solver as a physics-based surrogate to obtain the main characteristics of a good design in a relatively fast iterative process. The case study of a runner design for a low-head Francis turbine indicates advantages of integrating two derivative-free optimization algorithms with different local- and global search capabilities.

In the last decades, there has been a huge evolution in the development and application of Derivative Free Optimization (DFO) techniques. One of the major emerging DFO application fields are the optimal design of industrial products and machines and the development of efficient computer software. This Ph.D. thesis focuses on the exploration of the emerging DFO techniques, oriented especially in the optimization of real-world industrial problem. In this thesis is firstly presented a brief overview of the main DFO techniques and application. Then, are reported two works describing the development of DFO approaches aimed to tackle the optimization of Computer Vision Algorithms (CVA), employed in the automatic defect detection of pieces produced by a real-world industries.

In this paper, we consider the solution of global optimization problems involving hidden constraints. We present a novel deterministic derivative-free global optimization algorithm based on our recently introduced DIRECT-GL (Stripinis et al. in Optim Lett. 12(7):1699–1712, 2018). The new algorithm (DIRECT-GLh) incorporates two additional techniques for faster feasibility detection and solution improvement, especially when the solution lies on the border of the a priori unknown feasible region. The algorithm's potential is demonstrated on 67 problems from the current release of DIRECTLib and 100 Emmental-type test problems. The numerical comparison reveals the proposed algorithm's superiority to the considered DIRECT-type approaches for such problems.

Steel-making production is a dynamic process that has the characteristics of high temperature and heat, and a complex reaction mechanism that causes the mechanism model possibly to be unavailable and the system to be a black box. In this article, a dynamic operation optimization (DOO) problem is refined from the basic oxygen furnace (BOF) steel-making process, and the system model is formulated by a data analytics method. This prevents to solve the optimization problem with derivative-based optimization methods. To circumvent these difficulties, a surrogate-model-based derivative-free optimization algorithm is proposed for solving the DOO problem. In order to establish the surrogate model with the least number of function evaluations, a subset selection strategy is designed to find a sparse structure for the optimization problem, based on which a set of simple bases is determined to establish the surrogate model. Moreover, this also reduces the scale of the parameter optimization problem. Numerical experiments on actual production data verify the applicability and effectiveness of the proposed method.

Two-dimensional benchmark cases involving lift-constrained drag minimization in inviscid and viscous transonic flows are solved using derivative-free multi-fidelity optimization algorithms (space mapping and manifold mapping) and are compared with direct gradient-based optimization algorithms using adjoint sensitivities and trust regions. With 8 B-spline design variables, the multi-fidelity algorithms yield optimized shapes comparable to the shapes obtained by the direct algorithms but at a fraction of the cost. In particular for the inviscid case, the multi-fidelity algorithms needed less than 150 equivalent high-fidelity model evaluations (only flow solutions) taking approximately 460 minutes on a HPC with 32 processors, whereas the direct algorithm needed 391 high-fidelity model evaluations (flow and adjoint) taking approximately 4,494 minutes on the same HPC. For the viscous case, the multi-fidelity algorithms yield an optimized shape using less than 125 equivalent high-fidelity evaluations taking approximately 17.4 hours on the HPC. The direct algorithm was unsuccessful in optimizing the baseline shape in this case. A simple variation of surrogate-based optimization, the sequential approximate optimization (SAO), is utilized to optimize the twist distribution of a rectangular unswept wing in inviscid flow. Using 3 Bspline design variables, the SAO algorithm is able to obtain an optimized design with a nearelliptic section lift distribution. The total optimization cost is 22 high-fidelity model evaluations or approximately 42.5 hours on a HPC with 32 processors.

Polymeric outdoor insulators in transmission and distribution systems are becoming more common due to their various advantages. However, it is essential to control the electric stress of the composite insulators because of their organic nature, which is affected significantly by electrical discharges. Corona ring is a very effective tool to control the electric field in critical regions. In this study, a 3d model for the damaged field-aged 400 kV polymeric insulator is developed using COMSOL using the finite element method. The insulator was fitted with two corona rings at the HV terminal and one at the ground terminal of the split type. The optimization of the corona rings near both end-fittings is conducted using the optimization feature in COMSOL Multiphysics 5.5. We used the gradient-free (derivative-free) optimization algorithm (BOBYQA) in the optimizing module in COMSOL. The various parameters (ring height, ring radius, and tube radius) of both corona rings near the high voltage and ground ends are optimized. The results are compared without and with corona rings of old and optimized design. Finally, the corona inception level is determined. The promising results obtained from simulations showed improved field performance of the 400 kV composite insulator.

In this work we present an approach to determine geologically plausible reservoir models that are consistent with well-log data and multimodal observations (e.g., production and seismic data). The methodology is based on optimization techniques that are not invasive to the simulators used to reproduce observations. The geologically plausible reservoir models obtained facilitate reliable probabilistic production forecasts and lead to more robust optimization and decision-making support tools. The methodology approaches model inversion as an optimization problem where the cost function is a measure of the discrepancy between simulated responses and measurement data. Reservoir model uncertainty quantification is addressed by means of multi-start optimization, i.e., the use of local optimization methods with multiple initial guesses. Parameter reduction techniques allow a more effective and efficient global exploration of the optimization space. Geological realism is incorporated in the approach through multiple-point statistics. The set of reservoir models, determined as the output of the multi-start optimization, is thereafter included in a framework for robust optimization under uncertainty (e.g., to determine the drilling location of a new well). The optimization algorithms used are noninvasive with respect to the simulators used in the workflow and are implemented on a distributed- computing cluster. Different aspects of this approach are illustrated by means of a comprehensive case example that uses well-log and production data. It is important to note that the use of simulator-noninvasive (black box, derivative-free) optimization algorithms reduces significantly the dependence of the methodology on the simulators. The optimization methods tested include Generalized Pattern Search, Mesh Adaptive Direct Search, Sequential Quadratic Programming (with numerical derivatives) and Particle Swarm Optimization, a population-based procedure. These algorithms may yield efficient performance in terms of wall-clock time when they are combined with parameter reduction techniques and distributed computing (as is the case in this work). When parallel computing resources are scarce or nonexistent one may resort to methods suited to serial implementations (examples of these simulator-noninvasive algorithms are Hooke-Jeeves Direct Search and model-based approaches). The parameter reduction techniques studied here are based on Principal Component Analysis. In particular, the recently introduced Optimization-based Principal Component Analysis has been observed to deal with complex geological models satisfactorily. Uncertainty in future production is then incorporated in the robust optimization of well placement and of reactive well controls. The Ensemble Kalman Filter is widely used in industry for reservoir model uncertainty quantification. This method fails in some occasions (e.g., the ensemble collapses) for reasons that are not yet well understood. The multistart methodology proposed in this work can be an alternative to the Ensemble Kalman Filter in these problematic situations. Seismic data could be included in this methodology to mitigate uncertainty in probabilistic forecasts.