A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

First Order Bayesian Optimization (FOBO) is a sample efficient sequential approach to find the global maxima of an expensive-to-evaluate black-box objective function by suitably querying for the function and its gradient evaluations. Such methods assume Gaussian process (GP) models for both, the function and its gradient, and use them to construct an acquisition function that identifies the next query point. In this paper, we propose a class of practical FOBO algorithms that efficiently utilizes the information from the gradient GP to identify potential query points with zero gradients. We construct a multi-level acquisition function where in the first step, we optimize a lower level acquisition function with multiple restarts to identify potential query points with zero gradient value. We then use the upper level acquisition function to rank these query points based on their function values to potentially identify the global maximum. As a final step, the potential point of maximum is chosen as the actual query point. We validate the performance of our proposed algorithms on several test functions and show that our algorithms outperform state-of-the-art FOBO algorithms. We also illustrate the application of our algorithms in finding optimal set of hyper-parameters in machine learning and in learning the optimal policy in reinforcement learning tasks.

We propose a nested weighted Tchebycheff Multi-objective Bayesian optimization (WTB MOBO) framework where we built a regression model selection procedure from the ensemble of models, toward better estimation of the uncertain parameters (utopia) of the weighted Tchebycheff expensive black-box multi-objective function. In our previous work, a weighted Tchebycheff MOBO approach has been demonstrated which attempts to estimate the model parameters (utopia) in formulating the acquisition function of the weighted Tchebycheff multi-objective black-box functions, through calibration using an a priori selected regression model. However, the existing MOBO model lacks flexibility in selecting the appropriate regression models given the guided sampled data and, therefore, can under-fit or over-fit as the iterations of the MOBO progress. This ultimately can reduce the overall MOBO performance. As, in general, it is too complex to a priori guarantee a best model, this motivates us to consider a portfolio of different families (simple-to-complex) of predictive models that have been fitted with current training data guided by the WTB MOBO, and the best model is selected following a user-defined prediction root-mean-square error-based approach. The proposed approach is implemented in optimizing a thin tube design under constant loading of temperature and pressure, minimizing the risk of creep-fatigue failure and design cost. Finally, the nested WTB MOBO model performance is compared with different MOBO frameworks with respect to accuracy in parameter estimation, Pareto-optimal solutions, and function evaluation cost. This approach is generalized enough to consider different families of predictive models in the portfolio for best model selection, where the overall design architecture allows for solving any high-dimensional (multiple functions) complex black-box problems and can be extended to any other global criterion multi-objective optimization methods where prior knowledge of utopia is required.

A Surrogate-Based Optimization Method with Dynamic Adaptation for High-Dimensional Mixed-Integer Problems

Direct zigzag search for discrete multi-objective optimization

The presence of black-box functions in engineering design, which are usually computation-intensive, demands efficient global optimization methods. This work proposes a new global optimization method for black-box functions. The global optimization method is based on a novel mode-pursuing sampling (MPS) method which systematically generates more sample points in the neighborhood of the function mode while statistically covers the entire search space. Quadratic regression is performed to detect the region containing the global optimum. The sampling and detection process iterates until the global optimum is obtained. Through intensive testing, this method is found to be effective, efficient, robust, and applicable to both continuous and discontinuous functions. It supports simultaneous computation and applies to both unconstrained and constrained optimization problems. Because it does not call any existing global optimization tool, it can be used as a standalone global optimization method for inexpensive problems as well. Limitation of the method is also identified and discussed.

In this paper, we propose the StepDIRECT algorithm for derivative-free optimization (DFO), in which the black-box objective function has a stepwise landscape. Our framework is based on the well-known DIRECT algorithm. By incorporating the local variability to explore the flatness, we provide a new criterion to select the potentially optimal hyper-rectangles. In addition, we introduce a stochastic local search algorithm performing on potentially optimal hyper-rectangles to improve the solution quality and convergence speed. Global convergence of the StepDIRECT algorithm is provided. Numerical experiments on optimization for random forest models and hyper-parameter tuning are presented to support the efficacy of our algorithm. The proposed StepDIRECT algorithm shows competitive performance results compared with other state-of-the-art baseline DFO methods including the original DIRECT algorithm.

Fuel cell trucks offer significant environmental and performance advantages such as zero emissions, fast hydrogen refueling, and low noise, making them suitable for long-distance heavy-load transportation. Reducing hydrogen consumption to save operating costs has consistently been a critical focus in fuel cell truck research. To reduce hydrogen consumption and improve driving range, this paper proposes a fuzzy logic control method with uncertain rules leveraging digital twin technology. First, a digital twin model of the fuel cell truck is established, incorporating uncertain rule-based fuzzy logic control. Next, we set the parameters of the fuzzy logic controller as decision variables and define the hydrogen consumption of the fuel cell truck during the controlled period as the objective function, which is a black-box objective function. Furthermore, we employ an algorithm combining the surrogate model-based algorithm and the adaptive particle swarm optimization algorithm to solve the constructed optimization problem and determine the optimal fuzzy logic control parameters. Finally, the effectiveness and superiority of the proposed digital twin-based fuzzy logic control method with uncertain rules are verified through simulations and hardware-in-the-loop experiments.

Information-based Bayesian optimization (BO) algorithms have achieved state-of-the-art performance in optimizing a black-box objective function. However, they usually require several approximations or simplifying assumptions (without clearly understanding their effects on the BO performance) and/or their generalization to batch BO is computationally unwieldy, especially with an increasing batch size. To alleviate these issues, this paper presents a novel trusted-maximizers entropy search (TES) acquisition function: It measures how much an input query contributes to the information gain on the maximizer over a finite set of trusted maximizers, i.e., inputs optimizing functions that are sampled from the Gaussian process posterior belief of the objective function. Evaluating TES requires either only a stochastic approximation with sampling or a deterministic approximation with expectation propagation, both of which are investigated and empirically evaluated using synthetic benchmark objective functions and real-world optimization problems, e.g., hyperparameter tuning of a convolutional neural network and synthesizing ‘physically realizable’ faces to fool a black-box face recognition system. Though TES can naturally be generalized to a batch variant with either approximation, the latter is amenable to be scaled to a much larger batch size in our experiments.

Optimisation of linear structures is a well-developed and highly relevant research field. In particular, for linear problems with differentiable objective functions, gradient-based methods are well suited. In engineering practice, structural nonlinear behaviour and imperfections different from ideal design often need to be considered. Optimised lightweight structures, such as long-span bridges and stadium roofs, are more sensitive to sudden failure through buckling. Imperfections or perturbations in geometry and loading can drastically reduce the buckling load of a structure compared to that of a perfect structure. It is, however, computationally expensive to analyse structural problems with nonlinearities and imperfections and to compute the derivatives of the respective objective functions, rendering gradient-based methods unsuitable. In the present work, we use a Bayesian optimisation technique with a Gaussian process as the surrogate model for the described structural problems with random imperfections. Bayesian optimisation is exceptionally well suited for expensive black-box objective functions with no analytical expressions. To rapidly determine the random buckling loads, we consider an extended system of equilibrium equations for directly computing the buckling loads, i.e. bifurcation and snap-through points on the load-displacement path. This approach is exceedingly efficient for repeatedly computing buckling loads for different imperfections. We use the complex-step derivative approximation to determine the required directional derivatives of matrices. A low discrepancy sequence, Sobol sequence, is adopted for sampling from imperfections

We describe a method for multivariate function approximation which combines neural network learning, clustering and multiple regression. Neural networks with a single hidden layer are universal function approximators. However, due to the complexity of the network topology and the nonlinear transfer function used in computing the activation of the hidden units, the predictions of a trained network are difficult to comprehend. On the other hand, predictions from a multiple linear regression equation are easy to understand but not accurate when the underlying relationship between the input variables and the output variable as nonlinear. The method presented in this paper generates a set of multiple linear regression equations using neural networks. The number of regression equations as determined by clustering the weighted input variables. The predictions for samples in the same cluster are computed by the same regression equation. Experimental results on real-world data demonstrate that the new method generates relatively few regression equations from the training data samples. The errors an prediction using these equations are comparable to or lower than those achieved by existing function approximation methods.

A novel technique is developed for nonlinear optimization problem which is convex, separable and having multiple objective functions. In the development of the model all the objectives and the constraints of the multi objective model are linearly approximated over suitable intervals. The linear approximations are then aggregated to account for the original problem. The developed technique has been utilized for portfolio optimization problem. Firstly, the minimum variance model has been formulated and solved with machine leaning techniques. Secondly, the risk aversion model has been formulated and solved. The results obtained are combined into a multi objective framework of convex separable programming problem. All the three problems have been solved with the help of the XGBoost, neural network, and decision forest regression models. The renowned Python machine libraries of scikit-learn and keras have been utilized. The results identified portfolios that can return more financial benefits to the investors while investing in the capital market. The results of the proposed MOCSP approach are 22.5% improved in case of risk aversion model. Additionally, 17% improvement has been recorded in case of the minimum risk model. The MAE and RMSE for both XGBoost and decision forest regression have a frail value 0.0001. MAE and RMSE for the neural network regression have been recorded 1% and 2%, respectively. Both Accuracy and F1 score for XGBoost are 91%, for neural network regression are 98%, and for decision forest are 92%, respectively.

To non-destructively and rapidly monitor the chlorophyll content of winter wheat leaves under CO2 microleakage stress, and to establish the quantitative relationship between chlorophyll content and sensitive bands in the winter wheat growing season from 2023 to 2024, the leakage rate was set to 1 L/min, 3 L/min, 5 L/min, and 0 L/min through field experiments. The dimensional reduction was realized, fractional differential processing of a wheat canopy spectrum was carried out, a multiple linear regression (MLR) and partial least squares regression (PLSR) estimation model was constructed using a SPA selection band, and the model's accuracy was evaluated. The optimal model for hyperspectral estimation of wheat SPAD under CO2 microleakage stress was screened. The results show that the spectral curves of winter wheat leaves under CO2 microleakage stress showed a "red shift" of the green peak and a "blue shift" of the red edge. Compared with 1 L/min and 3 L/min, wheat leaves were more affected by CO2 at 5 L/min. Evaluation of the accuracy of the MLR and PLSR models shows that the MLR model is better, where the MLR estimation model based on 1.1, 1.8, 0.4, and 1.7 differential SPAD is the best for leakage rates of 1 L/min, 3 L/min, 5 L/min, and 0 L/min, with validation set R2 of 0.832, 0.760, 0.928, and 0.773, which are 11.528, 14.2, 17.048, and 37.3% higher than the raw spectra, respectively. This method can be used to estimate the chlorophyll content of winter wheat leaves under CO2 trace-leakage stress and to dynamically monitor CO2 trace-leakage stress in crops.

Quantitative identification of the saline soil salinity content is a necessary precondition for the reasonable improvement and utilization of saline land, the article aimed at comparing the different quantitative analysis methods and achieving fast estimation of the saline soil salt content in the Yellow River Delta based on the visible-near infrared spectroscopy. Kenli County in Shandong Province was selected as the experimental area, firstly, the representative soil samples were selected, hyperspectral reflectance of the soil samples were measured in situ and transformed to the first deviation. Secondly the correlate spectra, the characteristic spectra and indices were firstly filtered using correlation analysis. Finally, the estimation models of soil salinity content were built using the multiple linear regression (MLR), back propagation neural network (BPNN) and support vector machine (SVM) respectively. The results indicated that the characteristic wave bands of soil salinity were 684 nm and 2058 nm. On the condition of the same input variables, the prediction precision of the SVM models was the highest, followed by the BPNN, the MLR was the lowest. The SVM model based on the first deviation of the reflectance at 684 and 2058nm had the highest precision, with the calibration R2 of 0.91 and RMSE as 0.11%, the validation R2 of 0.93, RMSE as 0.26% and RPD as 2.61, which had very good prediction accuracy of soil salt content, and was very stable and reliable. Different input variables had a great impact on the model accuracy, among of the MLR models, only the precision of the model based on characteristic spectral indices was slightly higher and could be used to estimate salt content, among of the BPNN and SVM models, the precision of the models based on characteristic spectra and indices was more high and stable significantly than the models on the correlate spectra. Therefore, for the three modeling methods of multiple linear regression, back propagation neural network and support vector machine, building the estimation model of saline soil salinity content based on characteristic spectra indices was effective.

Estimation of black-box functions often requires evaluating an extensive number of expensive noisy points. Learning algorithms can actively compare the similarity between the evaluated and unevaluated points to determine the most informative subsequent points for efficient estimation of expensive functions in a sequential procedure. In this paper, we propose an active learning methodology based on the integration of Laplacian regularization and active learning - Cohn (ALC) measure for identification of the most informative points for efficient estimation of noisy black-box functions using Gaussian processes. We propose two simple greedy search algorithms for sequential optimization of the tuning parameters and determination of subsequent points based on the information from the previously evaluated points. We also enhance the graph Laplacian with the information of both the predictor and response variables to capture the similarity between the points more effectively. The proposed methodology is particularly suited for problems involving estimation of expensive black-box functions with a high level of noise and plenty of unevaluated points. Using a case study for analysis of the kinematics of pitching in baseball as well as simulation experiments, we demonstrate the performance of the proposed methodology against existing methods in the literature in terms of estimation error.