Bayesian Optimized Echo State Network Applied to Short-Term Load Forecasting

Quantitative analysis of heavy metals in soil by X-ray fluorescence with improved variable selection strategy and bayesian optimized support vector regression

Electrochemical machining is a promising non-traditional manufacturing process to make high-quality parts. The benefits of minimal thermally and mechanically induced stresses, free of burr, and a low surface roughness are appealing for industry and research institutes. However, the combined chemical reaction, electric field, fluid mechanics, and material properties involve a significant number of independent parameters which are difficult to analyze in order to draw comprehensive conclusions. To our current knowledge, process responses such as the material removal rate, optimal feed rate, and cutting profile cannot be represented accurately by analytical solutions. In recent years, deep learning has had tremendous success in analyzing sophisticated systems. The improved computation efficiency and reduced size of the training dataset required for deep learning have enabled various prediction models in the manufacturing industry. In this paper, a new approach is developed using the deep convolutional network with the Bayesian optimization algorithm to predict the diameters of the drilled hole from an electrochemical machining process. The Keras application programming interface (API) was used to build the deep convolutional network; the feed rate, pulse-on time, and voltage were used as input parameters to provide a fair comparison with a neural network from previous research. Random dropout layers were added to prevent overfitting of the network. Instead of tuning the network parameter by trial and error, the Bayesian parameter optimization algorithm was implemented to find the optimal set of parameters of the deep convolutional network that yields the minimum mean square error. The proposed algorithm was compared with a previously developed neural network with partially embedded physical knowledge. Improved training speed and accuracy were observed in comparison with the traditional neural network. The prediction model using the proposed deep learning algorithm demonstrated better prediction accuracy and provided a more systematic way to select the hyperparameter for the deep convolutional network.

As a novel recurrent neural network (RNN), an echo state network (ESN) that utilizes a reservoir with many randomly connected internal units and only trains the readout, avoids increased complexity of training procedures faced by traditional RNN. The ESN can cope with complex nonlinear systems because of its dynamical properties and has been applied in hydrological forecasting and load forecasting. Due to the linear regression algorithm usually adopted by generic ESN to train the output weights, an ill-conditioned solution might occur, degrading the generalization ability of the ESN. In this study, the ESN with Bayesian regularization (BESN) is proposed for short-term power production forecasting of small hydropower (SHP) plants. According to the Bayesian theory, the weights distribution in space is considered and the optimal output weights are obtained by maximizing the posterior probabilistic distribution. The evidence procedure is employed to gain optimal hyperparameters for the BESN model. The recorded data obtained from the SHP plants in two different counties, located in Yunnan Province, China, are utilized to validate the proposed model. For comparison, the feed-forward neural networks with Levenberg-Marquardt algorithm (LM-FNN) and the generic ESN are also employed. The results indicate that BESN outperforms both LM-FNN and ESN.

Echo state network (ESN), which efficiently models nonlinear dynamic systems, has been proposed as a special form of recurrent neural network. However, most of the proposed ESNs consist of complex reservoir structures, leading to excessive computational cost. Recently, minimum complexity ESNs were proposed and proved to exhibit high performance and low computational cost. In this paper, we propose a simple deterministic ESN with a loop reservoir, i.e., an ESN with an adjacent-feedback loop reservoir. The novel reservoir is constructed by introducing regular adjacent feedback based on the simplest loop reservoir. Only a single free parameter is tuned, which considerably simplifies the ESN construction. The combination of a simplified reservoir and fewer free parameters provides superior prediction performance. In the benchmark datasets and real-world tasks, our scheme obtains higher prediction accuracy with relatively low complexity, compared to the classic ESN and the minimum complexity ESN. Furthermore, we prove that all the linear ESNs with the simplest loop reservoir possess the same memory capacity, arbitrarily converging to the optimal value.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

In this study, an efficient optimization framework was developed to determine the parameters of through‐silicon vias and the layout of heat sources in three‐dimensional integrated circuits (3D ICs), employing an artificial neural network (ANN) reduced‐order model in conjunction with a Bayesian optimization (BO) algorithm. The proposed method effectively predicts the temperature distribution in 3D ICs and refines their thermal parameters, offering solutions to thermal management challenges. Latin hypercube sampling was utilized for data sampling, enhancing the previously established rapid thermal analysis method through parameterization of heat source locations. The temperature distribution data for varying hotspot locations in 3D ICs were fitted using an appropriately defined objective function, leading to the development of a reduced‐order ANN model that accelerates temperature prediction. The computational results demonstrate that the neural network model exhibits a deviation in predicted values of less than 2%, and the coefficient of determination R2 approximately 0.93, underscoring high predictive accuracy. Additionally, the optimization outcomes and the efficiency of the selected BO algorithm were thoroughly evaluated. Notably, the BO algorithm achieved the global optimum in just 4.07 s across 250 iterations, demonstrating an effective power distribution strategy for the 3D ICs model.

To address the problems of local optima and insufficient convergence accuracy in parameter identification of primary frequency regulation (PFR) for steam turbines, this paper proposed a hybrid identification method that integrated an Improved Bayesian Optimization (IBO) algorithm and an Improved Whale Optimization Algorithm (IWOA). By initializing the Bayesian parameter population using Tent chaotic mapping and the reverse learning strategy, employing a radial basis kernel function hyperparameter training mechanism based on the Adam optimizer and optimizing the Expected Improvement (EI) function using the Limited-memory Broyden–Fletcher– Goldfarb–Shanno with Bounds (L-BFGS-B) method, IBO was proposed to obtain the optimal candidate set with the smallest objective function value. By introducing a nonlinear convergence factor and the adaptive Levy flight perturbation strategy, IWOA was proposed to obtain locally optimized optimal solutions. By using the reverse-guided optimization mechanism and employing a fitness-oriented selection strategy, the optimal solution was chosen to complete the closed-loop process of reverse learning feedback. Nine standard test functions and the Proportional Integral Derivative (PID) parameter identification of the electro-hydraulic servo system in a 330 MW steam turbine were presented as examples. Compared with Particle Swarm Optimization (PSO), Whale Optimization Algorithm (WOA), Bayesian Optimization (BO) and Particle Swarm Optimization-Grey Wolf Optimizer (PSO-GWO), the Improved Bayesian Optimization-Whale Optimization Algorithm (IBO-WOA) proposed in this paper has been validated to effectively avoid the problem of getting stuck in local optima during complex optimization and has high parameter recognition accuracy. Meanwhile, an Out-Of-Distribution (OOD) Test based on noise injection had demonstrated that IBO-WOA had good robustness. The time constant identification of the steam turbine were carried out using IBO-WOA under two experimental conditions, and the identification results were input into the PFR model. The simulated power curve can track the experimental measured curve well, proving that the parameter identification results obtained by IBO-WOA have high accuracy and can be used for the modeling and response characteristic analysis of the steam turbine PFR.

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

A two-stage online remaining useful life prediction framework for supercapacitors based on the fusion of deep learning network and state estimation algorithm

Lithology identification during drilling plays a vital role in geological and geotechnical exploration, as it facilitates the early detection of formation-related hazards and supports the development of optimized mining strategies. Traditional lithology identification research involves problems such as fuzzy indicator characteristics and unbalanced sample quantities, which affect the accuracy and interpretability of model identification. In order to solve these problems, the Shanxi Guoqiang Coal Mine was taken as the research object, and a combined machine learning model was used to conduct a study on lithology identification during drilling. First, the least absolute shrinkage and selection operator (LASSO) algorithm was used to screen the independent variables and retain the parameters that contributed the most to lithology identification. Then, the synthetic minority oversampling technique (SMOTE) algorithm was used to expand the data samples, increase the amounts of minority sample data, and keep the ratios of various lithology data at 1:1. Then, the Bayesian optimization (BO) algorithm was used to optimize the penalty factor C and kernel function hyperparameter γ—two important parameters of the support vector machine (SVM) model—and the BO-SVM lithology identification model was established. Finally, the data samples were processed, and the results were compared with those of single models and unbalanced sample processing to evaluate their effect. The results showed the following: during the drilling process, the four indicators of drilling speed, mud pressure, slurry flow rate, and torque are strongly correlated with the lithology and can be used for lithology identification and classification research. After the data set was oversampled using the SMOTE algorithm, each model had better robustness and generalization ability; the classification result evaluation indicators were also greatly improved, especially for the random forest model, which had a poor original evaluation effect. The BO algorithm was used to optimize the parameters of the SVM model and establish a combined model that correctly identified 95 groups of data out of 96 groups of test samples with an identification accuracy rate of 99%, which was better than that of the traditional machine learning model. The evaluation results were compared with measured data, which confirmed the reliability of the combined model classification method and its potential to be extended to lithology identification and classification work.

Traditional machine learning algorithms have problems such as overfitting, low accuracy, and difficulty in hyperparameter optimization when performing fault diagnosis.In order to improve the accident diagnosis ability of nuclear power plant reactor system, this paper combines Bayesian optimization (BO) algorithm with eXtreme Gradient Boosting (XGBoost) algorithm to develop a reactor accident diagnosis model.First, data preprocessing and feature quantity analysis are performed on accident data samples.Then, the BO algorithm is used to optimize the hyperparameters of the XGBoost model. Finally, the BO-XGBoost model is used to diagnose the operating conditions of seven nuclear power plants, and the diagnostic effects of various traditional machine learning classification algorithms are compared and analyzed.The results show that the BO-XGBoost model can achieve more efficient and accurate identification of reactor accident types, and the model has better generalization ability.This research can help nuclear power plant operators to accurately identify the types of reactor accidents, assist decision-making, and ensure the safe operation of nuclear power plants.

<div>With the rapid development of autonomous driving technology, unmanned ground vehicles (UGVs) are gradually replacing humans to perform tasks such as reconnaissance, target tracking, and search in special scenarios. Omnidirectional mobility based on rapid adjustment of vehicle heading posture enhances the applicability of UGVs in specialized scenarios. Omnidirectional mobility signifies the capability for rapid adjustments to the vehicle’s heading angle, longitudinal velocity, and lateral velocity. Traditional vehicles are constrained by the limitations of under-actuation, which prevents active regulation of lateral movement. Instead, they rely on the coordinated regulation of longitudinal and yaw movements, failing to meet the requirements for omnidirectional mobility. Distributed vehicles featuring steering distributed between the front/rear axles and four-wheel independent drive leverage the over-actuation advantages provided by multi-actuator coordinated control, making them particularly suitable for omnidirectional mobility at large sideslip angles. This feature enables the UGVs to achieve rapid adjustment of vehicle heading posture. However, existing control strategies centered on stabilizing yaw rate and suppressing sideslip angles cannot adapt to the decoupling control requirements of such platforms. Additionally, the strong coupling characteristics between actuator subsystems further exacerbate control difficulties. To this end, this article proposes a full-state decoupling motion control strategy, the nonlinear model is locally linearized at each equilibrium point of the vehicle, and a set of equilibrium state models is derived. The validity of this local linearization method is verified through phase diagram analysis and modal analysis. The Bayesian optimization (BO) algorithm is then employed to optimize and identify the cornering stiffness of the front/rear axles at each equilibrium point in these locally linearized models, thereby enhancing the characterization ability of the linear model for the nonlinear dynamic model at the corresponding equilibrium points. Subsequently, a full-state decoupling motion controller is designed by integrating the model predictive control (MPC) algorithm. Finally, the controller presented in this article is employed on the distributed vehicle experiment platform (DVEP). The experimental results demonstrate that in two drift-like scenarios with different sideslip angles, compared with the baseline controller, the path tracking error of this method is reduced by more than 13%, and the sideslip angle tracking error is reduced by more than 12%.</div>

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Comparison of echo state network and feed-forward neural networks in electrical load forecasting for demand response programs

Diamond grinding wheels (DGWs) have a central role in cutting-edge industries such as aeronautics or defense and spatial applications. Characterizations of DGWs are essential to optimize the design and machining performance of such cutting tools. Thus, the critical issue of DGW characterization lies in the detection of diamond grits. However, the traditional diamond detection methods rely on manual operations on DGW images. These methods are time-consuming, error-prone and inaccurate. In addition, the manual detection of diamond grits remains challenging even for a subject expert. To overcome these shortcomings, we introduce a deep learning approach for automatic diamond grit segmentation. Due to our small dataset of 153 images, the proposed approach leverages transfer learning techniques with pre-trained ResNet34 as an encoder of U-Net CNN architecture. Moreover, with more than 8600 hyperparameter combinations in our model, manually finding the best configuration is impossible. That is why we use a Bayesian optimization algorithm using Hyperband early stopping mechanisms to automatically explore the search space and find the best hyperparameter values. Moreover, considering our small dataset, we obtain overall satisfactory performance with over 53% IoU and 69% F1-score. Finally, this work provides a first step toward diamond grinding wheel characterization by using a data-driven approach for automatic semantic segmentation of diamond grits.