Bayesian Optimized Deep Convolutional Network for Electrochemical Drilling Process

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Material removal rate optimization with bayesian optimized differential evolution based on deep learning in robotic polishing

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

Object detection systems based on the deep convolutional neural network (CNN) have recently made ground- breaking advances on several object detection benchmarks. While the features learned by these high-capacity neural networks are discriminative for categorization, inaccurate localization is still a major source of error for detection. Building upon high-capacity CNN architectures, we address the localization problem by 1) using a search algorithm based on Bayesian optimization that sequentially proposes candidate regions for an object bounding box, and 2) training the CNN with a structured loss that explicitly penalizes the localization inaccuracy. In experiments, we demonstrated that each of the proposed methods improves the detection performance over the baseline method on PASCAL VOC 2007 and 2012 datasets. Furthermore, two methods are complementary and significantly outperform the previous state-of-the-art when combined.

Quantitative analysis of heavy metals in soil by X-ray fluorescence with improved variable selection strategy and bayesian optimized support vector regression

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

To address the problems of local optima and insufficient convergence accuracy in parameter identification of primary frequency regulation (PFR) for steam turbines, this paper proposed a hybrid identification method that integrated an Improved Bayesian Optimization (IBO) algorithm and an Improved Whale Optimization Algorithm (IWOA). By initializing the Bayesian parameter population using Tent chaotic mapping and the reverse learning strategy, employing a radial basis kernel function hyperparameter training mechanism based on the Adam optimizer and optimizing the Expected Improvement (EI) function using the Limited-memory Broyden–Fletcher– Goldfarb–Shanno with Bounds (L-BFGS-B) method, IBO was proposed to obtain the optimal candidate set with the smallest objective function value. By introducing a nonlinear convergence factor and the adaptive Levy flight perturbation strategy, IWOA was proposed to obtain locally optimized optimal solutions. By using the reverse-guided optimization mechanism and employing a fitness-oriented selection strategy, the optimal solution was chosen to complete the closed-loop process of reverse learning feedback. Nine standard test functions and the Proportional Integral Derivative (PID) parameter identification of the electro-hydraulic servo system in a 330 MW steam turbine were presented as examples. Compared with Particle Swarm Optimization (PSO), Whale Optimization Algorithm (WOA), Bayesian Optimization (BO) and Particle Swarm Optimization-Grey Wolf Optimizer (PSO-GWO), the Improved Bayesian Optimization-Whale Optimization Algorithm (IBO-WOA) proposed in this paper has been validated to effectively avoid the problem of getting stuck in local optima during complex optimization and has high parameter recognition accuracy. Meanwhile, an Out-Of-Distribution (OOD) Test based on noise injection had demonstrated that IBO-WOA had good robustness. The time constant identification of the steam turbine were carried out using IBO-WOA under two experimental conditions, and the identification results were input into the PFR model. The simulated power curve can track the experimental measured curve well, proving that the parameter identification results obtained by IBO-WOA have high accuracy and can be used for the modeling and response characteristic analysis of the steam turbine PFR.

A two-stage online remaining useful life prediction framework for supercapacitors based on the fusion of deep learning network and state estimation algorithm

Deep (machine) learning in recent years has significantly increased the predictive modeling strengths for applications in the areas of computer vision, speech recognition, and natural language processing. Being heavily data-driven, conventional general-purpose deep learning methods mainly rely on training data to learn task-specific data representations. In this thesis, we focus on leveraging novel learning algorithms in deep learning to tackle visual recognition challenges at the three key levels -- namely pixel, object, and image levels. Throughout the thesis, we demonstrate how each and every of our proposed methods can address a specific visual recognition problem. Firstly, we apply deep learning to improve the saliency detection performance of deep networks. A coarse saliency map of the input image's salient object is initially predicted by a convolutional-deconvolutional network (or equivalently a fully convolutional network). Following that, the coarse saliency map is sequentially refined by a separate network called Recurrent Attentional Convolutional-Deconvolutional Network (RACDNN) that shares its network weights across its sequential steps. At every sequential (temporal) step, the same RACDNN is applied to a newly selected/attended image sub-region and it performs saliency map refinement for that particular sub-region. Despite having many learnable network parameters and no access to additional training data, RADCNN can be trained well and it outperforms the "single-step" (coarse saliency detection) baseline and state-of-the-art saliency detection methods by large margins, on several saliency detection benchmark datasets. Secondly, we introduce auto-encoder weight transfer networks (AE-WTN+), a deep model for scaling up object detection. Conventional WTN transfers object class knowledge from the classification weights of a pretrained large-scale image classification network, to the classification weights of an object detection network. However, they suffer from under-fitting and are not able to preserve the rich object class information of novel classes. In this work, we propose a new WTN method -- AE-WTN+ which makes use of normalization techniques to ease network optimization. Inputs and intermediate features are normalized to improve trainability for reduced under-fitting. More importantly, a novel auto-encoding (reconstruction) loss is proposed to encourage information preservation of all classes in the weight predictions of AE-WTN+. This enables the predicted weights of AE-WTN+ to maintain the class discriminability and neighbourhood relationships contained in the pre-trained large-scale classification weights. Experiments on large-scale detection datasets validate the effectiveness of AE-WTN+ on both seen and novel classes. Thirdly, we improve the efficiency of cross-layer network connections for very deep convolutional network in the context of image classification. This gives rise to a new convolutional network architecture called Deluge Networks (DelugeNets) which pass the output (feature activations) of any preceding composite network layer as direct inputs to all succeeding composite layers. For the incoming feature activations of preceding composite layers, the succeeding composite layers first apply rather lightweight cross-layer depthwise convolutional weights on them for feature aggregation. The heavy lifting is carried out by the last convolutional weight layer in each preceding composite layer that performs cross-channel convolutions. The proposed technique enables unimpeded information flow among many layers in very deep DelugeNets with great efficiency. Experiments on small-scale and large-scale datasets show that DelugeNets perform competitively to state-of-the-art deep convolutional networks at lower computational and parameter costs. Fourthly, we propose Stochastic Downsampling (SDPoint), a method to overcome a major limitation suffered by conventional deep convolutional networks -- their computational costs during inference are fixed and cannot be changed. SDPoint downsamples the feature maps at a randomly selected point (layer index) in the network hierarchy during training. The many downsampling configurations known as SDPoint instances each entail a unique downsampling point and downsampling ratio. The same convolutional network weights are shared across the different SDPoint instances that downsample feature maps differently. During inference, the computational cost of a network trained with SDPoint can be instantaneously adjusted to fit a given computational budget, by manually selecting an appropriate SDPoint instance. Additionally, sharing weights between different feature map scales provides significant regularization benefits, making the networks trained with SDPoint less sensitive to scales and more competent. In image classification experiments, SDPoint demonstrates significant cost-accuracy and computational advantages, over independently-trained (non-weight-shared) baselines with various inference costs.

In this study, an efficient optimization framework was developed to determine the parameters of through‐silicon vias and the layout of heat sources in three‐dimensional integrated circuits (3D ICs), employing an artificial neural network (ANN) reduced‐order model in conjunction with a Bayesian optimization (BO) algorithm. The proposed method effectively predicts the temperature distribution in 3D ICs and refines their thermal parameters, offering solutions to thermal management challenges. Latin hypercube sampling was utilized for data sampling, enhancing the previously established rapid thermal analysis method through parameterization of heat source locations. The temperature distribution data for varying hotspot locations in 3D ICs were fitted using an appropriately defined objective function, leading to the development of a reduced‐order ANN model that accelerates temperature prediction. The computational results demonstrate that the neural network model exhibits a deviation in predicted values of less than 2%, and the coefficient of determination R2 approximately 0.93, underscoring high predictive accuracy. Additionally, the optimization outcomes and the efficiency of the selected BO algorithm were thoroughly evaluated. Notably, the BO algorithm achieved the global optimum in just 4.07 s across 250 iterations, demonstrating an effective power distribution strategy for the 3D ICs model.

Computer-aided diagnosis of ocular disorders like glaucoma can be effectively performed with retinal fundus images. Since the advent of machine learning and later deep learning techniques in medical image analysis, research focusing on automated and early detection of glaucoma has gained prominence. In this paper, we show how a deep Convolutional Neural Network model can be used in visualising the features related to Glaucoma that are consistent with (a subset of) those used by ophthalmologists for diagnosis. However, since the doctors do not necessarily depend entirely on these features but back them up with other important measurements such as visual field testing (HVF), any deep neural network such as CNN may not necessarily be expected to yield 100% accuracy. Further, other important parameters such as Intraocular Pressure (IOP), which may be built into a deep learning model, may not necessarily correlate well with the presence of Glaucoma. In this work, a deep CNN model has been trained and tested on a large number of high-quality fundus images, both normal and glaucomatous. We compare the results with transfer learning models such as VGG16, ResNet50, and MobileNetV2. On an average, we obtained an accuracy of 93.75% in identifying glaucoma, focusing only on the features of the fundus image.

In this study, a novel method for detecting the growth of delamination in sandwich structures has been proposed. To this end, we suggested hybridizing the Deep Learning techniques (DL) and Finite Element Method (FEM) for predicting the growth of delamination in this structures. A dataset of simulated delamination growth under different delamination sizes has been produced using the FEM method. Then, a DL model has been trained using this dataset to precisely predict the growth of delamination. This study focused on predicting delamination growth using a tuned and optimized deep learning based regressor. Therefore, to find the ideal set of hyperparameters, the Bayesian optimization algorithm has been used for selecting the best structure and enhancing the regressor performance. Afterward, the model was evaluated and multiple processes were conducted to improve its behavior and solve its stability and overfitting issues. Particularly, an inconsistency between validation loss and training loss has been initially detected in the behaviour of the model, which may indicate overfitting. To tackle this issue, dropout regularization has been added, which improved the consistency between the loss functions but results in less smooth convergence from the expectations. So, in a third study, dropout and L1 regularization has been combined to improve the stability of the model. This combination achieved a consistent and smooth convergence between the validation and training loss functions. The findings highlight the importance of hyperparameter optimization and regularization techniques in improving regression model performance. The study shows the efficiency of Bayesian optimization in hyperparameter tuning and the iterative optimization of a regression model. Furthermore, the outcomes show that the suggested method can identify and predict delamination growth with high accuracy.

Diabetic Maculopathy (DM) is considered the most common cause of permanent visual impairment in diabetic patients. The absence of clear pathological symptoms of DM hinders the timely diagnosis and treatment of such a critical condition. Early diagnosis of DM is feasible through eye screening technologies. However, manual inspection of retinography images by eye specialists is a time-consuming routine. Therefore, many deep learning-based computer-aided diagnosis systems have been recently developed for the automatic prognosis of DM in retinal images. Manual tuning of deep learning network’s hyperparameters is a common practice in the literature. However, hyperparameter optimization has shown to be promising in improving the performance of deep learning networks in classifying several diseases. This study investigates the impact of using the Bayesian optimization (BO) algorithm on the classification performance of deep learning networks in detecting DM in retinal images. In this research, we propose two new custom Convolutional Neural Network (CNN) models to detect DM in two distinct types of retinal photography; Optical Coherence Tomography (OCT) and fundus retinography datasets. The Bayesian optimization approach is utilized to determine the optimal architectures of the proposed CNNs and optimize their hyperparameters. The findings of this study reveal the effectiveness of using the Bayesian optimization for fine-tuning the model hyperparameters in improving the performance of the proposed CNNs for the classification of diabetic maculopathy in fundus and OCT images. The pre-trained CNN models of AlexNet, VGG16Net, VGG 19Net, GoogleNet, and ResNet-50 are employed to be compared with the proposed CNN-based models. Statistical analyses, based on a one-way analysis of variance (ANOVA) test, receiver operating characteristic (ROC) curve, and histogram, are performed to confirm the performance of the proposed models.

To reduce patient's dose, few-view CT reconstruction promises to be a good attempt. The key to better reconstruction is the sparse view artifacts. In recent years, DL(deep learing) has attracted a lot of attention because its outstanding performance in image processing. We propose a deep learning method for few-view CT reconstuction. Our method directly learns an end-to-end mapping between the full-view/few-view reconstruction. The mapping is represented as a deep convolutional neural network (CNN) that takes the few-view reconstruction image as the input and outputs the full-view one. We further show that traditional Dictionary Learning based reconstruction methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art reconstruction quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed.

Local binary features for texture classification: Taxonomy and experimental study