Recent Advances in Bayesian Optimization

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

To address the problems of local optima and insufficient convergence accuracy in parameter identification of primary frequency regulation (PFR) for steam turbines, this paper proposed a hybrid identification method that integrated an Improved Bayesian Optimization (IBO) algorithm and an Improved Whale Optimization Algorithm (IWOA). By initializing the Bayesian parameter population using Tent chaotic mapping and the reverse learning strategy, employing a radial basis kernel function hyperparameter training mechanism based on the Adam optimizer and optimizing the Expected Improvement (EI) function using the Limited-memory Broyden–Fletcher– Goldfarb–Shanno with Bounds (L-BFGS-B) method, IBO was proposed to obtain the optimal candidate set with the smallest objective function value. By introducing a nonlinear convergence factor and the adaptive Levy flight perturbation strategy, IWOA was proposed to obtain locally optimized optimal solutions. By using the reverse-guided optimization mechanism and employing a fitness-oriented selection strategy, the optimal solution was chosen to complete the closed-loop process of reverse learning feedback. Nine standard test functions and the Proportional Integral Derivative (PID) parameter identification of the electro-hydraulic servo system in a 330 MW steam turbine were presented as examples. Compared with Particle Swarm Optimization (PSO), Whale Optimization Algorithm (WOA), Bayesian Optimization (BO) and Particle Swarm Optimization-Grey Wolf Optimizer (PSO-GWO), the Improved Bayesian Optimization-Whale Optimization Algorithm (IBO-WOA) proposed in this paper has been validated to effectively avoid the problem of getting stuck in local optima during complex optimization and has high parameter recognition accuracy. Meanwhile, an Out-Of-Distribution (OOD) Test based on noise injection had demonstrated that IBO-WOA had good robustness. The time constant identification of the steam turbine were carried out using IBO-WOA under two experimental conditions, and the identification results were input into the PFR model. The simulated power curve can track the experimental measured curve well, proving that the parameter identification results obtained by IBO-WOA have high accuracy and can be used for the modeling and response characteristic analysis of the steam turbine PFR.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Key-frame extraction is a key technique of video retrieval. In view of the low accuracy of currently traditional key-frame extraction algorithm, and low recall level and accuracy of video retrieval, a kind of model of network video retrieval of dynamic Bayesian network optimization for the mobile terminal was put forward. First of all, the study of dynamic Bayesian network optimization algorithm was conducted, and by using a Bayesian algorithm, the real data were used as the characteristic of the center of clustering, then the condensed coarsening and the condensed refining that are two important processes for network video retrieval model algorithm were improved, used to replace the top core set obtained in the original coarsening process, thereby designing a kind of new dynamic Bayesian network optimization algorithm, so as to realize the rapid and accurate positioning of the top core set, which can decrease the number of condensed layer and simplify the computational complexity of the algorithm. Then, the improved network video retrieval model algorithm was applied to the key-frame extraction of the video. The experimental results show that the proposed algorithm can more effectively extract key frames of video compared with the original algorithm.

Electrochemical machining is a promising non-traditional manufacturing process to make high-quality parts. The benefits of minimal thermally and mechanically induced stresses, free of burr, and a low surface roughness are appealing for industry and research institutes. However, the combined chemical reaction, electric field, fluid mechanics, and material properties involve a significant number of independent parameters which are difficult to analyze in order to draw comprehensive conclusions. To our current knowledge, process responses such as the material removal rate, optimal feed rate, and cutting profile cannot be represented accurately by analytical solutions. In recent years, deep learning has had tremendous success in analyzing sophisticated systems. The improved computation efficiency and reduced size of the training dataset required for deep learning have enabled various prediction models in the manufacturing industry. In this paper, a new approach is developed using the deep convolutional network with the Bayesian optimization algorithm to predict the diameters of the drilled hole from an electrochemical machining process. The Keras application programming interface (API) was used to build the deep convolutional network; the feed rate, pulse-on time, and voltage were used as input parameters to provide a fair comparison with a neural network from previous research. Random dropout layers were added to prevent overfitting of the network. Instead of tuning the network parameter by trial and error, the Bayesian parameter optimization algorithm was implemented to find the optimal set of parameters of the deep convolutional network that yields the minimum mean square error. The proposed algorithm was compared with a previously developed neural network with partially embedded physical knowledge. Improved training speed and accuracy were observed in comparison with the traditional neural network. The prediction model using the proposed deep learning algorithm demonstrated better prediction accuracy and provided a more systematic way to select the hyperparameter for the deep convolutional network.

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Revolutionizing biochar synthesis for enhanced heavy metal adsorption: Harnessing machine learning and Bayesian optimization

A two-stage online remaining useful life prediction framework for supercapacitors based on the fusion of deep learning network and state estimation algorithm

In this study, an efficient optimization framework was developed to determine the parameters of through‐silicon vias and the layout of heat sources in three‐dimensional integrated circuits (3D ICs), employing an artificial neural network (ANN) reduced‐order model in conjunction with a Bayesian optimization (BO) algorithm. The proposed method effectively predicts the temperature distribution in 3D ICs and refines their thermal parameters, offering solutions to thermal management challenges. Latin hypercube sampling was utilized for data sampling, enhancing the previously established rapid thermal analysis method through parameterization of heat source locations. The temperature distribution data for varying hotspot locations in 3D ICs were fitted using an appropriately defined objective function, leading to the development of a reduced‐order ANN model that accelerates temperature prediction. The computational results demonstrate that the neural network model exhibits a deviation in predicted values of less than 2%, and the coefficient of determination R2 approximately 0.93, underscoring high predictive accuracy. Additionally, the optimization outcomes and the efficiency of the selected BO algorithm were thoroughly evaluated. Notably, the BO algorithm achieved the global optimum in just 4.07 s across 250 iterations, demonstrating an effective power distribution strategy for the 3D ICs model.

We propose a highly efficient searching algorithm in crystal structure prediction. The searching algorithm is a hybrid of the evolutionary algorithm and Bayesian optimization. The evolutionary algorithm is widely used in crystal structure prediction, and the Bayesian optimization is one of the selection-type algorithms we have developed. We have performed simulations of crystal structure prediction to compare the success rates of the random search, evolutionary algorithm, Bayesian optimization, and hybrid algorithm for up to ternary systems such as Si, Y2Co17, Al2O3, and CuGaS2, using the CrySPY code. These results demonstrate that the evolutionary algorithm can generate structures more efficiently than random structure generation, and the Bayesian optimization can efficiently select potential candidates from a large number of structures. Moreover, the hybrid algorithm, which has the advantages of both, is proved to be the most efficient searching algorithm among them.

Bayesian Co-evolutionary Optimization based entropy search for high-dimensional many-objective optimization

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

In recent years, the complex international environment and economic situation have made soybean futures prices increasingly unstable, which is not conducive to financial stability. Therefore, this paper uses a BO-CNN-LSTM model to accurately predict soybean futures prices and to manage price fluctuations for investors and governments. Firstly, LSTM network is employed to predict soybean futures prices using the local features extracted by CNN network. In addition, CNN-LSTM hyperparameters are optimally solved using Bayesian optimization algorithms. Finally, the constructed model is compared with BP neural network, LSTM model and CNN-LSTM model. This paper selects the basic daily data of the soybean futures contract No.1 of Dalian Commodity Exchange from 2014 to 2021 for research. According to the results, CNN-LSTM models based on Bayesian optimization algorithms perform best. Compared with the basic CNN-LSTM model, MAPE increased by 44.17%, RMSE increased by 24.61%, MAE increased by 41.48%, and R2 increased by 0.06%, which demonstrates Bayesian optimization's superiority.

With accelerating climate change and urbanization, regional carbon balance faces increasing uncertainty. Terrestrial carbon sinks play a crucial role in advancing China’s sustainable development under the dual-carbon strategy. This study quantitatively modeled China’s terrestrial carbon sink capacity and analyzed the multidimensional relationships between impact factors and carbon sinks. After preprocessing multi-source raster data, we introduced kernel normalized the difference vegetation index (kNDVI) to the Carnegie–Ames–Stanford approach (CASA) model, together with a heterotrophic respiration (Rh) empirical equation, to simulate pixel-level net ecosystem productivity (NEP) across China. A light gradient-boosting machine (LightGBM) model, optimized via Bayesian algorithms, was trained to regress NEP drivers, categorized into atmospheric components (O3, NO2, and SO2) and subsurface properties (a digital elevation model (DEM), enhanced vegetation index (EVI), soil moisture (SM)), and human activities (land use/cover change (LUCC), POP, gross domestic product (GDP)). Shapley Additive Explanation (SHAP) values were used for model interpretation. The results reveal significant spatial heterogeneity in NEP across geographic and climatic contexts. The pixel-level mean and total NEP in China were 268.588 gC/m2/yr and 2.541 PgC/yr, respectively. The north tropical zone (NRZ) exhibited the highest average NEP (828.631 gC/m2/yr), while the middle subtropical zone (MSZ) and south subtropical zone (SSZ) demonstrated the most stable NEP distributions. LightGBM achieved high simulation accuracy, further enhanced by Bayesian optimization. SHAP analysis identified EVI as the most influential factor, followed by SM, NO2, DEM, and POP. Additionally, LightGBM effectively captured nonlinear relationships and variable interactions.