Automatic Approximation of Expensive Functions with Active Learning

Global sensitivity analysis (GSA) and uncertainty quantification (UQ) for groundwater modeling are challenging because of the model complexity and significant computational requirements. To reduce the massive computational cost, a cheap‐to‐evaluate surrogate model is usually constructed to approximate and replace the expensive groundwater models in the GSA and UQ. Constructing an accurate surrogate requires actual model simulations on a number of parameter samples. Thus, a robust experimental design strategy is desired to locate informative samples so as to reduce the computational cost in surrogate construction and consequently to improve the efficiency in the GSA and UQ. In this study, we develop a Taylor expansion‐based adaptive design (TEAD) that aims to build an accurate global surrogate model with a small training sample size. TEAD defines a novel hybrid score function to search informative samples, and a robust stopping criterion to terminate the sample search that guarantees the resulted approximation errors satisfy the desired accuracy. The good performance of TEAD in building global surrogate models is demonstrated in seven analytical functions with different dimensionality and complexity in comparison to two widely used experimental design methods. The application of the TEAD‐based surrogate method in two groundwater models shows that the TEAD design can effectively improve the computational efficiency of GSA and UQ for groundwater modeling.

When dealing with computationally expensive simulation codes or process measurement data, global surrogate modeling methods are firmly established as facilitators for design space exploration, sensitivity analysis, visualization and optimization. Popular surrogate model types include neural networks, support vector machines, and splines. In addition, the cost of each simulation mandates the use of active learning strategies where data points (simulations) are selected intelligently and incrementally. When applying surrogate models to multi-output systems, the hyperparameter optimization problem is typically formulated in a single objective way. The different response outputs are modeled separately by independent models. Instead, a multi-objective approach would benefit the domain expert by giving information about output correlation, facilitate the generation of diverse ensembles, and enable automatic model type selection for each output on the fly. This paper outlines a multi-objective approach to surrogate model generation including its application to two problems.KeywordsPareto FrontMultiobjective OptimizationKriging ModelDesign Space ExplorationBrute Force SearchThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Surrogate modeling of hydrodynamic forces between multiple floating bodies through a hierarchical interaction decomposition

Adaptive surrogate-assisted multi-objective evolutionary algorithm using an efficient infill technique

Many real-world applications can be formulated as expensive multimodal optimization problems (EMMOPs). When surrogate-assisted evolutionary algorithms (SAEAs) are employed to tackle these problems, they not only face the problem of selecting surrogate models but also need to tackle the problem of discovering and updating multiple modalities. Different optimization problems and different stages of evolutionary algorithms (EAs) generally require different types of surrogate models. To address this issue, in this article, we present a multisurrogate-assisted multitasking particle swarm optimization algorithm to seek multiple optimal solutions of EMMOPs at a low computational cost. The proposed algorithm first transforms an EMMOP into a multitasking optimization problem by integrating various surrogate models, and designs a multitasking niche particle swarm algorithm to solve it. Following that, a surrogate model management strategy based on the skill factor and clustering is developed to effectively balance the number of real function evaluations and the prediction accuracy of candidate optimal solutions. In addition, an adaptive local search strategy based on the trust region is proposed to enhance the capability of swarm in exploiting potential optimal modalities. We compare the proposed algorithm with five state-of-the-art SAEAs and seven multimodal EAs on 19 benchmark functions and the building energy conservation problem and experimental results show that the proposed algorithm can obtain multiple highly competitive optimal solutions.

Previous studies have suggested the use of reduced-order models calibrated by means of high-fidelity load simulations as means for computationally inexpensive wind turbine load assessments; the so far best performing surrogate modelling approach in terms of balance between accuracy and computational cost has been the polynomial chaos expansion (PCE). Regarding the growing interest in advanced machine learning applications, the potential of using Artificial Neural-Network (ANN) based surrogate models for improved simplified load assessment is investigated in this study. Different ANN model architectures have been evaluated and compared to other types of surrogate models (PCE and quadratic response surface). The results show that a feedforward neural network with two hidden layers and 11 neurons per layer, trained with the Levenberg Marquardt backpropagation algorithm is able to estimate blade root flapwise damage-equivalent loads (DEL) more accurately and faster than a PCE trained on the same data set. Further research will focus on further model improvements by applying different training techniques, as well as expanding the work with more load components.

Landslides are dangerous disasters that are affected by many factors. Neural networks can be used to fit complex observations and predict landslide displacement. However, hyperparameters have a great impact on neural networks, and each evaluation of a hyperparameter requires the construction of a corresponding model and the evaluation of the accuracy of the hyperparameter on the test set. Thus, the evaluation of hyperparameters requires a large amount of time. In addition, not all features are positive factors for predicting landslide displacement, so it is necessary to remove useless and redundant features through feature selection. Although the accuracy of wrapper-based feature selection is higher, it also requires considerable evaluation time. Therefore, in this paper, reliability-enhanced surrogate-assisted particle swarm optimization (RESAPSO), which uses the surrogate model to reduce the number of evaluations and combines PSO with the powerful global optimization ability to simultaneously search the hyperparameters in the long short-term memory (LSTM) neural network and the feature set for predicting landslide displacement is proposed. Specifically, multiple surrogate models are utilized simultaneously, and a Bayesian evaluation strategy is designed to integrate the predictive fitness of multiple surrogate models. To mitigate the influence of an imprecise surrogate model, an intuitional fuzzy set is used to represent individual information. To balance the exploration and development of the algorithm, intuition-fuzzy multiattribute decision-making is used to select the best and most uncertain individuals from the population for updating the surrogate model. The experiments were carried out in CEC2015 and CEC2017. In the experiment, RESAPSO is compared with several well-known and recently proposed SAEAs and verified for its effectiveness and advancement in terms of accuracy, convergence speed, and stability, with the Friedman test ranking first. For the landslide displacement prediction problem, the RESAPSO-LSTM model is established, which effectively solves the feature selection and LSTM hyperparameter optimization and uses less evaluation time while improving the prediction accuracy. The experimental results show that the optimization time of RESAPSO is about one-fifth that of PSO. In the prediction of landslide displacement in the step-like stage, RESAPSO-LSTM has higher prediction accuracy than the contrast model, which can provide a more effective prediction method for the risk warning of a landslide in the severe deformation stage.

The paper shows the feasibility of building closed-form and fast-to-evaluate surrogate models via supervised kernel-based machine learning regressions behaving as digital twins for computationally expensive multibody simulations. The aforementioned surrogate models are adopted to predict the railway vehicle dynamics safety indexes defined in the international standards, depending on the wheel-rail forces, directly from the results of longitudinal train dynamics simulations. The digital twin models are trained with the outputs of Simpack multibody simulations of a reference freight wagon, to which the in-train forces calculated by an in-house MATLAB longitudinal train dynamics simulator are applied. Two machine learning techniques are considered: the support vector machine and the least-squares support vector machine regressions. Both techniques ensure a good accuracy even with a limited number of training samples. The derivation of the surrogate models can strongly enhance the modelling capabilities of common longitudinal train dynamics simulators, that cannot evaluate the wheel-rail contact forces. At the same time, the method shown in the paper allows to strongly reduce the total computational times, as the evaluation of the closed-form surrogate models allows to effectively estimate the safety indexes with no need for computationally demanding multibody simulations.

Examination of artificial neural networks to predict wind-induced displacements of cable net roofs

In surrogate evolutionary algorithms, usually the type of surrogate model is chosen beforehand, and it is never changed during the run of the evolution. Moreover, the reasoning why a particular type of model was chosen is often missing. In this paper, we present a framework which in each generation selects the most suitable surrogate from a set of models based on some pre-defined criteria. The results based on different types of model selectors are compared, and the dynamics of the evolution together with the change of the selected model type during the run of the evolutionary algorithm are discussed.

Surrogate models are a cost-effective replacement for expensive computer simulations in design space exploration. Literature has already demonstrated the feasibility of accurate surrogate models for single radio frequency (RF) and microwave devices. Within the European Marie Curie project O-MOORE-NICE! (Operational Model Order Reduction for Nanoscale IC Electronics) we aim to investigate the feasibility of the surrogate modeling approach for entire RF circuit blocks. This paper presents an overview about the surrogate model type selection problem for low noise amplifier modeling.

Surrogate modeling immersed probability density evolution method for structural reliability analysis in high dimensions

In this paper, an effort has been made to maximize the efficiency of ESP by using surrogate models for optimization. Inlet and outlet blade angles are selected as design variables. Initially, a pump stage simulation is performed using Navier Stokes solver which include rotating impeller and stationary diffuser. The steps of the simulations are: generate geometry, grid and solve. The surrogate models were response surface approximation, Kriging and radial basis neural network. A hybrid genetic algorithm has been implemented to find optimal design. The multiple surrogate approach was used to reduce uncertainty in optimal point search. The initial CFD generated results were used to generate surrogates and the surrogates were used to give optimal point via genetic algorithm. Design of experiments were used to find the designs and low fidelity based surrogates were constructed to reduce the number of simulation runs. The optimization procedure sufficiently reduces the design time. Hence, the approach shows an optimal design of ESP and selection method of right surrogate for the shortening of computational cost.

The paper introduces a novel modality to efficiently tune the convolutional layers of a deep neural network (CNN) and an approach to also rank the importance of the involved hyperparameters. Evolutionary algorithms (EA) offer a flexible solution to this twofold issue, while the expensive simulations of the deep learner with the generated configurations are resolved by surrogate modelling. Three models have been used and evaluated as surrogates: random forests (RF), support vector machines (SVM) and Kriging. Sample convolutional configurations are generated by Latin hypercube sampling and have attached computed accuracy outcomes from real CNN runs. For the hyperparameter estimation task, the fitness of an individual from the EA associated with a surrogate model is subsequently derived from the CNN accuracy estimation on those variable values. With respect to the ranking and variable selection task, RF includes implicit variable selection, the SVM can be straightforwardly supported by a second EA, and Kriging offers a ranking based on the corresponding θ values. The estimated accuracy of the found hyperparameter values is compared with the true validation accuracy, and they are next used for the prediction on the test cases. The ranking of the variables for each of the three surrogate models is compared, and their influence is also revealed by response surface methodology. The experimental testing of the proposed EA–surrogate approaches is conducted on a real-world scenario of histopathological image interpretation in colorectal cancer diagnosis.

In electronic circuit design, preliminary analyses of the circuit performances are generally carried out using time-consuming simulations. These analyses should be performed as fast as possible because of the strict temporal constraints on the industrial sector time to market. On the other hand, there is the need of precision and reliability of the analyses. For these reasons, there is more and more interest toward surrogate models able to approximate the behavior of a device with a high precision making use of a limited set of samples. Using suitable surrogate models instead of simulations, it is possible to perform a reliable analysis in less time. In this work, we are going to analyze how the surrogate models given by the support vector machine (SVM) perform when they are used to approximate the behavior of industrial circuits that will be employed in consumer electronics. The SVM is also compared to the surrogate models given by the response surface methodology using a commercial software currently adopted for this kind of applications.