An Improved AdaBoost Algorithm for Hyperparameter Optimization

To understand the change trend of Lake Taihu’s water quality indexes in advance and prevent the occurrence of pollution incidents such as blue algae, the ammonia nitrogen index forecasted model was studied by taking the water quality monitoring data of Nanquan in Wuxi City as an example. The chromaticity, odor and taste, pH, nitrite nitrogen, chloride, dissolved oxygen, and algae were determined as input variables by implementing linear regression analysis through SPSS software to establish the Extreme Learning Machine (ELM) model for forecasting ammonia nitrogen index. Aiming at the problem of ELM randomly generating weights and thresholds resulting in low forecast accuracy, the Particle Swarm Optimization (PSO) algorithm was adopted to optimize ELM parameters. In order to solve the situation that the PSO algorithm is easy to fall into local optimum, the Adaboost algorithm was introduced to train weak ELM predictors, which were later combined into a strong predictor for the forecast. The results represent that the Adaboost-PSO-ELM model’s forecast coefficient of determination (\({R}^{2}\)) and root mean square error (\(RMSE\)) for ammonia nitrogen are 0.955 mg/L and 0.023 mg/L, respectively. Compared with the two models of ELM and PSO-ELM, the forecast accuracy is significantly improved. Therefore, the ELM model based on Adaboost and PSO algorithms is feasible for forecasting water quality index of ammonia nitrogen in Lake Taihu, providing new ideas for accurate forecast in water quality aspect.KeywordsLake TaihuAmmonia nitrogenAdaboost-PSO-ELM

Recently, focus on maintenance strategies has been shifted towards prognostic health management (PHM) and a number of state of the art algorithms based on data-driven prognostics have been developed to predict the health states of degrading components based on sensory data. Amongst these algorithms, Multiclass Support Vector Machines (MC-SVM) has gained popularity due to its relatively high classification accuracy, ability to classify multiple patterns and capability to handle noisy / incomplete data. However, its application is limited by the difficulty in determining the required kernel function and penalty parameters. To address this problem, this paper proposes a hybrid differential evolution – particle swarm optimization (DE-PSO) algorithm to optimize the MC-SVM kernel function and penalty parameters. The differential algorithm (DE) obtains the search limit for the SVM parameters, while the particle swarm optimization algorithm (PSO) determines the global optimum parameters for a given training data set. Since degrading machinery components display several degradation stages in their lifetime, the MC-SVM trained with optimum parameters are used to estimate the health states of a degrading machinery component, from which the remaining useful life (RUL) is predicted. This method improves the classification accuracy of MC-SVM in predicting the health states of a machinery component and consequently increases the accuracy of RUL predictions. The feasibility of the method is validated using bearing prognostic run-to-failure data obtained from NASA public data repository. A comparative study between MC-SVM with parameters obtained using simple grid search with n-fold cross validation and MC- SVM with DE-PSO based on prognostic performance metrics reveals that the proposed method has better performance, with all the cases considered falling within a 10 % error margin. The method also outperforms other soft computing methods proposed in literature.

High accuracy remote sensed image classification technology is a long-term and continuous pursuit goal of remote sensing applications. In order to evaluate single classification algorithm accuracy, take Landsat TM image as data source, Northwest Yunnan as study area, seven types of land cover classification like Maximum Likelihood Classification has been tested, the results show that: (1)the overall classification accuracy of Maximum Likelihood Classification(MLC), Artificial Neural Network Classification(ANN), Minimum Distance Classification(MinDC) is higher, which is 82.81% and 82.26% and 66.41% respectively; the overall classification accuracy of Parallel Hexahedron Classification(Para), Spectral Information Divergence Classification(SID), Spectral Angle Classification(SAM) is low, which is 37.29%, 38.37, 53.73%, respectively. (2) from each category classification accuracy: although the overall accuracy of the Para is the lowest, it is much higher on grasslands, wetlands, forests, airport land, which is 89.59%, 94.14%, and 89.04%, respectively; the SAM, SID are good at forests classification with higher overall classification accuracy, which is 89.8% and 87.98%, respectively. Although the overall classification accuracy of ANN is very high, the classification accuracy of road, rural residential land and airport land is very low, which is 10.59%, 11% and 11.59% respectively. Other classification methods have their advantages and disadvantages. These results show that, under the same conditions, the same images with different classification methods to classify, there will be a classifier to some features has higher classification accuracy, a classifier to other objects has high classification accuracy, and therefore, we may select multi sub-classifier integration to improve the classification accuracy. © (2015) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

The article describes problems that cannot always be solved by machine learning algorithms to get an accurate result. For such tasks, you need to use ensembles that actually combine several algorithms in order to simultaneously learn and correct errors, thereby improving the accuracy of results at times, compared to other algorithms. Here are described high accuracy and stability of ensembles, since they are extremely important for fast processing of a large amount of data and at the same time a fairly accurate result. Today, ensembles actively compete with neural networks, since both are quite effective for the same tasks. The work is devoted to the study of ensembles, and more specifically Adaptive Boosting. The main ensemble methods and their advantages and disadvantages were considered. Here are examined the basic principles of the AdaBoost boosting ensemble, analyzes the industries where this ensemble is already used. The study discusses ways for ensembles to work: stacking, running, and boosting. This work analyzes the boosting algorithms such as AdaBoost and gradient boosting. The advantages and disadvantages of these algorithms are presented. This work presents the principle of operation of the AdaBoost algorithm and provides an example of its operation. An analytical review of the AdaBoost algorithm is also given. Examples of using AdaBoost are given. The AdaBoost algorithm was experimentally applied to the data set and its effectiveness was tested. The effectiveness of ensemble boosting methods that compete with neural networks is analyzed. As a result, it is proved that they are somewhat more stable and understandable, which is why they get an advantage in the choice. Also, the AdaBoost algorithm can slightly improve the result compared to “strong” classifiers. Here are presented prospects for applying the AdaBoost algorithm, which is still being developed and will be used in further research.KeywordsMachine learningArtificial intelligenceDecision treeAdaBoostStackingBaggingBoostingAlgorithmClustering

There is no single best Hyperparameter Optimization algorithm. Different optimization algorithms meet different Hyperparameter Optimization tasks with different constraints. To accelerate the Hyperparameter Optimization, it is necessary to parallelize training executions of various tests, introduce distributed training and prematurely stop unpromising tests. It is recommended to use a library approach to support the hyperparameter optimization service. Ray Tune is the best by far among the open source Hyperparameter Optimization. Hyperparameter optimisation (HPO) is the process of identifying a set of hyperparameters that yields an optimal model. By optimal, we mean the model that minimises a predefined loss function on a given data set. This is a repeated model training process, except that the neural network is trained with a different set of hyperparameters each time. In this process, the optimal set of hyperparameters will be identified. During the grid search, users specify a limited set of values for each hyperparameter and then select trial hyperparameters from the Cartesian product of these values. Once the grid is constructed, the GPGs are tested with the grid values. The grid search suffers when the number of hyperparameters becomes larger or the parameter search space becomes larger, as in this case the required number of estimates will grow exponentially. Another problem with grid search is its inefficiency. Since grid search treats each set of hyperparameter candidates equally, it will spend a lot of computational resources in the suboptimal configuration space without spending enough computational power on the optimal space.

Market intelligence to crowded event is critical in decision making to advertising campaigns and support planning. In this paper describes gender classification and age estimation based on hyperparameter tuning optimization Computer Vision and Machine Learning Approach. The images of Myanmar People are quite challenging with existing famous IMDBs dataset due to different ethnics and nations. The proposed system presented to achieve improving better analysis with using hyperparameter optimization by Deep Learning Fast-R-CNN method on three Datasets, IMDBs, Asia Image Dataset and new Myanmar Image Dataset. Specially, this system adapted fast-R-CNN with hyper parameter optimization using Nelder-Mead methods. Experiences result demonstrated that hybrid method outperformed than other method and achieved the state-of-art-accurate results on real-world Myanmar Video streaming to achieve target audience.

Hyper-Parameter Optimization (HPO) occupies a fundamental role in Deep Learning systems due to the number of hyper-parameters (HPs) to be set. The state-of-the-art of HPO methods are Grid Search, Random Search and Bayesian Optimization. The first two methods try all possible combinations and random combination of the HPs values, respectively. This is performed in a blind manner, without any information for choosing the new set of HPs values. Bayesian Optimization (BO), instead, keeps track of past results and uses them to build a probabilistic model mapping HPs into a probability density of the objective function. Bayesian Optimization builds a surrogate probabilistic model of the objective function, finds the HPs values that perform best on the surrogate model and updates it with new results. In this paper, we improve BO applied to Deep Neural Network (DNN) by adding an analysis of the results of the network on training and validation sets. This analysis is performed by exploiting rule-based programming, and in particular by using Probabilistic Logic Programming. The resulting system, called Symbolic DNN-Tuner, logically evaluates the results obtained from the training and the validation phase and, by applying symbolic tuning rules, fixes the network architecture, and its HPs, therefore improving performance. We also show the effectiveness of the proposed approach, by an experimental evaluation on literature and real-life datasets.

BackgroundShock wave lithotripsy (SWL), ureteroscopy, and percutaneous nephrolithotomy are established treatments for renal stones. Historically, SWL has been a predominant and commonly used procedure for treating upper tract renal stones smaller than 20 mm in diameter due to its noninvasive nature. However, the reported failure rate of SWL after one treatment session ranges from 30% to 89%. The failure rate can be reduced by identifying candidates likely to benefit from SWL and manage patients who are likely to fail SWL with other treatment modalities. This would enhance and optimize treatment results for SWL candidates.ObjectiveWe proposed to develop a machine learning model that can predict SWL outcomes to assist practitioners in the decision-making process when considering patients for stone treatment.MethodsA data set including 58,349 SWL procedures performed during 31,569 patient visits for SWL to a single hospital between 1990 and 2016 was used to construct and validate the predictive model. The AdaBoost algorithm was applied to a data set with 17 predictive attributes related to patient demographics and stone characteristics, with success or failure as an outcome. The AdaBoost algorithm was also applied to a training data set. The generated model’s performance was compared to that of 5 other machine learning algorithms, namely C4.5 decision tree, naïve Bayes, Bayesian network, K-nearest neighbors, and multilayer perceptron.ResultsThe developed model was validated with a testing data set and performed significantly better than the models generated by the other 5 predictive algorithms. The sensitivity and specificity of the model were 0.875 and 0.653, respectively, while its positive predictive value was 0.7159 and negative predictive value was 0.839. The C-statistics of the receiver operating characteristic (ROC) analysis was 0.843, which reflects an excellent test.Conclusions We have developed a rigorous machine learning model to assist physicians and decision-makers to choose patients with renal stones who are most likely to have successful SWL treatment based on their demographics and stone characteristics. The proposed machine learning model can assist physicians and decision-makers in planning for SWL treatment and allow for more effective use of limited health care resources and improve patient prognoses.

<bold>Background</bold>Neural networks, with their powerful fitting capabilities, can learn the relationships between input and output variables based on large amounts of data, often serve as proxy models for physical programs in the field of engineering calculations, including nuclear engineering calculations. Neutron transport calculations, as one of the core links in neutronics simulations, are often suffer from lengthy computational times. However, this issue can also be addressed by utilizing neural network models. Nevertheless, neural network models have a series of hyperparameters that need to be set, but manually adjusting these hyperparameters is laborious, repetitive, and reliant only on experience. Moreover, these hyperparameters are not reusable when solving different problems.<bold>Purpose</bold>This study aims to propose a novel algorithm to improve computational efficiency of core neutronic parameter prediction, and provide reference for the application of artificial intelligence in core physics calculation.<bold>Methods</bold>The dataset was derived from the VITAS program's calculations of the TAKEDA1 and TAKEDA2 benchmark problems, consisting of 10 000 and 20 000 sets respectively. The core layout sequence was taken as input whilst the output included the effective multiplication factor and the regional integral flux. Thereafter, the core layout sequence was mapped into a one-dimensional vector, and the dataset was divided into training and validation sets in a 6:4 ratio. Then, the Bayesian optimization algorithm was employed to adjust fully connected neural network (FCNN) hyperparameters, and the adaptive learning rate decay and loss function optimization methods were combined to automatically search for the optimal combination of hyperparameters for datasets with different problems. Finally, critical core parameters, i.e., the effective multiplication factor and the regional integrated flux, were fitted to valid the generalization and accuracy of abovementioned algorithm, and a comparative experiment was conducted using manually set hyperparameters and hyperparameters output by Bayesian optimization as neural network training parameters.<bold>Results</bold>The results demonstrate that Bayesian optimization effectively enhances the neural network's accuracy. Specifically, the average error of the effective multiplication factor is within 1.50×10^-3, while that of the regional integral flux on the TAKEDA1 dataset is 1.72%, with a maximum error rate of 7.56%.<bold>Conclusions</bold>This approach can automatically search for the optimal combination of hyperparameters for different datasets to achieve the best performance, demonstrating high flexibility, efficiency, and strong generalization.

This research has three parts addressing classification, especially the imbalanced data problem, which is one of the most popular and essential issues in the domain of classification. The first part is to study the Adaptive Boosting (AdaBoost) algorithm. AdaBoost is an effective solution for classification, but it still needs improvement in the imbalanced data problem. This part proposes a method to improve the AdaBoost algorithm using the new weighted vote parameters for the weak classifiers. Our proposed weighted vote parameters are determined not only by the global error rate but also by the classification accuracy rate of the positive class, which is our primary interest. The imbalanced index of the data is also a factor in constructing our algorithms. The numeric studies show that our proposed algorithms outperform the traditional ones, especially regarding the evaluation criterion of the F--1 Measure. Theoretic proofs of the advantages of our proposed algorithms are presented. The second part treats the Relevance Vector Machine (RVM), which is a supervised learning algorithm extended from the Support Vector Machine (SVM) based on the Bayesian sparsity model. Compared with the regression problem, RVM classification is challenging to conduct because there is no closed-form solution for the weight parameter posterior. The original RVM classification algorithm uses Newton's method in optimization to obtain the mode of weight parameter posterior, then approximates it by a Gaussian distribution in Laplace's method. This original model would work, but it just applies the frequency methods in a Bayesian framework. This part first proposes a Generic Bayesian RVM classification, which is a pure Bayesian model. We conjecture that our algorithm achieves convergent estimates of the quantities of interest compared with the nonconvergent estimates of the original RVM classification algorithm. Furthermore, a fully Bayesian approach with the hierarchical hyperprior structure for RVM classification is proposed, which improves the classification performance, especially in the imbalanced data problem. The third part is an extended work of the second one. The original RVM classification model uses the logistic link function to build the likelihood, which makes the model hard to conduct since the posterior of the weight parameter has no closed-form solution. This part proposes the probit link function approach instead of the logistic one for the likelihood function in RVM classification, namely PRVM (RVM with the Probit link function). We show that the posterior of the weight parameter in our model follows the multivariate normal distribution and achieves a closed-form solution. A latent variable is needed in our algorithm to simplify the Bayesian computation greatly, and its conditional posterior follows a truncated normal distribution. Compared with the original RVM classification model, our proposed one is another pure Bayesian approach and it has a more efficient computation process. For the prior structure, we first consider the Normal-Gamma independent prior to propose a Generic Bayesian PRVM algorithm. Furthermore, the Fully Bayesian PRVM algorithm with a hierarchical hyperprior structure is proposed, which improves the classification performance, especially in the imbalanced data problem.

In this paper, a method of polarimetric SAR image classification based on polarimetric decomposition and AdaBoost algorithm is proposed. The proposed method improves classification accuracy and speed. AdaBoost algorithm, as a robust learner with high accuracy, can fully utilize the polarimetric features to achieve image classification. In simulated tests, the proposed method is observed to produce improved classification accuracy and speed, compared with H /α classification algorithm.

Transformation rate for electric energy meter is the premise of accurate metering. The meter for electric energy measurement in the distribution network provides the basic data for the calculation of the line loss. It is shown in survey that nearly 3% of the energy meters’ transformation rate in the database are different from their actual rate. An inaccuracy transformation rate of electric energy meter will cause errors in calculating power and the line loss rate of low-voltage distribution network. However, it is not easy to find out the inaccuracy transformation rate in a database in daily management. To solve this problem, this paper presents a method based on adaboost and BP neural networks, and proves the feasibility of the method. BP neural networks serve as the weak classifiers. The weights of the sample and the weights of the weak classifiers are calculated based on adaboost algorithm. In the BP neural network for classification, the sample weights are utilized to adjust the error function of BP neural network to optimize the classification. A strong classifier is combined with weighting output of the weak classifiers. Computing results show that this method can precisely pick out electric energy meter with inaccuracy rate based on the power energy data. Accuracy of BP-adaboost classification is higher than that of the BP neural network, and the training time is greatly shortened. This method can improve the efficiency of line loss management in low-voltage distribution networks and greatly reduce consuming the manpower and material resources.

Recent developments show that naive Bayesian classifier (NBC) performs significantly better in applications, although it is based on the assumption that all attributes are independent of each other. However, in the NBC each variable has a finite number of values, which means that in large data sets NBC may not be so effective in classifications. For example, variables may take continuous values. To overcome this issue, many researchers used fuzzy naive Bayesian classification for partitioning the continuous values. On the other hand, the choice of the distance function is an important subject that should be taken into consideration in fuzzy partitioning or clustering. In this study, a new fuzzy Bayes classifier is proposed for numerical attributes without the independency assumption. To get high accuracy in classification, membership functions are constructed by using the fuzzy C-means clustering (FCM). The main objective of using FCM is to obtain membership functions directly from the data set instead of consulting to an expert. The proposed method is demonstrated on the basis of two well-known data sets from the literature, which consist of numerical attributes only. The results show that the proposed the fuzzy Bayes classification is at least comparable to other methods.

Support Vector Regression (SVR) is one of the frequently used methods for regression estimation. Kernel selection in SVR is of great importance for the training and estimation of the model. Determining the parameters of the selected kernel is at least as important as the selected kernel. The two parameters of the Radial Basis Function (RBF) kernel are determined stochastically with 5 different current and popular meta-heuristic optimization algorithms and their performances are evaluated on 7 different data sets. The performances of Particle Swarm Optimization (PSO), Salp Swarm Algorithm (SSA), Kepler Optimization Algorithm (KOA), Weighted Average Algorithm (WAA) and Hippopotamus Optimization (HO) algorithms in hyper-parameter optimization are compared. Data sets are separated into training and test with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{k}$</tex>-10 cross-validation. WAA was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$0.8521,0.9493$</tex>, and 0.8392 in 3 of the 7 data sets, respectively. HO was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of 0.9374, and 0.9375 in 2 of the 7 datasets, respectively. The SVR model, which did not undergo hyper-parameter optimization with the optimization algorithm, was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 8 6 6 9}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 5 9 2 8}$</tex> in 2 of the 7 datasets. Among the algorithms performing hyperparameter optimization, KOA was found to be the fastest algorithm in terms of run time, while HO was found to be the slowest.

In this paper, a combination of artificial neural network (ANN)and genetic algorithm(GA) has been proposed as a method to obtain a high accuracy in classification of polarimetric SAR data. First we extracted 57 features based on decomposition algorithms and thenthe best features among inputted features by use of GA-ANN wereselected.The classification results of a data set, composed of different land cover elements, exhibited higher accuracy than maximum likelihood and Wishart classifier; moreover the input features were decreased to small numbers which contain sufficient information for classification of data set.