Semi-supervised Ensemble Learning Using Label Propagation

Multi-model ensemble learning for battery state-of-health estimation: Recent advances and perspectives

Order-up-to-level inventory optimization model using time-series demand forecasting with ensemble deep learning

The increasing number of space debris and space-active targets makes the space environment more and more complex. Space target recognition, a crucial component of space situational awareness, is of paramount importance to space security. Firstly, this paper elucidates the fundamental principles of ensemble learning, analyzes its characteristics and fusion method, and provides a comprehensive comparison of three common ensemble learning methods. Secondly, this paper analyzes the basic attributes and characteristics of space targets and categorizes the hierarchy of space target recognition. Again, the paper reviews the advances in the application of ensemble learning in space target recognition, focusing on three aspects: space target recognition datasets, the ensemble of traditional machine learning models, and ensemble deep learning. Subsequently, classical machine learning and ensemble learning algorithms are tested on a self-built space target simulation dataset, and we find that Stacking performs well on this dataset. Finally, the paper discusses future research directions.

The use of actual electricity consumption data provided the chance to detect the change of customer class types. This work could be done by using classification techniques. However, there are several challenges in computational techniques. The most important one is to efficiently handle a large number of dimensions to increase customer classification performance. In this paper, we proposed a symmetrical uncertainty based feature subset generation and ensemble learning method for the electricity customer classification. Redundant and significant feature sets are generated according to symmetrical uncertainty. After that, a classifier ensemble is built based on significant feature sets and the results are combined for the final decision. The results show that the proposed method can efficiently find useful feature subsets and improve classification performance.

In this research work, we have proposed a novel ensemble learning approach “BBS method” which stands for Bagging, Boosting and Stacking with appropriate base classifiers for the classification of the five UCI datasets taken from the field of Bioinformatics. Experiments are conducted using Weka and Java Eclipse and it has been observed empirically that our approach gives better accuracy with lower root mean square error rate using the technique of ensemble learning. Henceforth we conclude that our proposed ensemble learning method is more suitable in handling the classification problem in the bioinformatics domain. Such approaches can be efficiently used in related real-life scenarios of classification domain.

Perfume identification (PI) based on an electronic nose (EN) can be used for exposing counterfeit perfumes more time-efficiently and cost-effectively than using gas chromatography and mass spectrometry instruments. During the past five years, decision-tree-based ensemble learning methods, also called tree ensemble learning methods, have demonstrated excellent performance when solving multi-class classification problems. However, the performance of tree ensemble learning methods for the EN-based PI problem remains uncertain. In this paper, four well-known tree ensemble learning classification methods, random forest (RF), stagewise additive modeling using a multi-class exponential loss function (SAMME), gradient-boosting decision tree (GBDT), and extreme gradient boosting (XGBoost), were implemented for PI using our self-designed EN. For fair comparison, all the tested classification methods used as input the same feature data extracted using principal component analysis. Moreover, two benchmark methods, neural network and support vector machine, were also tested with the same experimental setup. The quantitative results of experiments undertaken demonstrated that the mean PI accuracy achieved by XGBoost was up to 97.5%, and that XGBoost outperformed other tested methods in terms of accuracy mean and variance based on our self-designed EN.

Diagnosing and treating at-risk patients for chronic kidney disease (CKD) relies heavily on accurately classifying the disease. The use of deep learning models in healthcare research is receiving much interest due to recent developments in the field. CKD has many features; however, only some features contribute weightage for the classification task. Therefore, it is required to eliminate the irrelevant feature before applying the classification task. This paper proposed a hybrid feature selection method by combining the two feature selection techniques: the Boruta and the Recursive Feature Elimination (RFE) method. The features are ranked according to their importance for CKD classification using the Boruta algorithm and refined feature set using the RFE, which recursively eliminates the least important features. The hybrid feature selection method removes the feature with a low recursive score. Later, selected features are given input to the proposed ensemble deep learning method for classification. The experimental ensemble deep learning model with feature selection is compared to Support Vector Machine (SVM), Logistic Regression (LR), and Random Forest (RF) models with and without feature selection. When feature selection is used, the ensemble model improves accuracy by 2%. Experimental results found that these features, age, pus cell clumps, bacteria, and coronary artery disease, do not contribute much to accurate classification tasks. Accuracy, precision, and recall are used to evaluate the ensemble deep learning model.

Material decomposition allows the reconstruction of material-specific images in spectral X-ray imaging, which requires efficient decomposition models. Due to the presence of nonideal effects in X-ray imaging systems, it is difficult to explicitly estimate the imaging systems for material decomposition tasks. As an alternative to previous empirical material decomposition methods, we investigated material decomposition using ensemble learning methods in this paper. Three ensemble methods with two decision trees as the base learning algorithms were investigated to perform material decomposition in both simulation and experiment. The results were quantitatively evaluated for comparison studies. In general, the results demonstrate that the proposed ensemble learning methods often outperform their base learning algorithms, and rarely reduce performance. Compared to the reference methods and its base learning algorithm, the performance of the Boosting method using REPTree with regularization is improved by over 42% and 13%, respectively, in the noiseless simulated scenario of the XCAT phantom with cardiac and respiratory motion, and over 36% and 17%, respectively, in the noisy scenario. Simultaneously, the performance is improved by over 9% and 8%, respectively, in the original torso phantom scenario, and over 13% and 12%, respectively, in the denoising scenario. The results indicate that ensemble learning with gradient descent optimization algorithms is more appropriate for material decomposition tasks. Further studies are encouraged to facilitate future efforts toward clinical applications.

Using an ensemble of neural networks with consistency regularization is effective for improving performance and stability of deep learning, compared to the case of a single network. In this paper, we present a semi-supervised Deep Coupled Ensemble (DCE) model, which contributes to ensemble learning and classification landmark exploration for better locating the final decision boundaries in the learnt latent space. First, multiple complementary consistency regularizations are integrated into our DCE model to enable the ensemble members to learn from each other and themselves, such that training experience from different sources can be shared and utilized during training. Second, in view of the possibility of producing incorrect predictions on a number of difficult instances, we adopt class-wise mean feature matching to explore important unlabeled instances as classification landmarks, on which the model predictions are more reliable. Minimizing the weighted conditional entropy on unlabeled data is able to force the final decision boundaries to move away from important training data points, which facilitates semi-supervised learning. Ensemble members could eventually have similar performance due to consistency regularization, and thus only one of these members is needed during the test stage, such that the efficiency of our model is the same as the non-ensemble case. Extensive experimental results demonstrate the superiority of our proposed DCE model over existing state-of-the-art semi-supervised learning methods.

The Internet of Things (IoT) integrates more than billions of intelligent devices over the globe with the capability of communicating with other connected devices with little to no human intervention. IoT enables data aggregation and analysis on a large scale to improve life quality in many domains. In particular, data collected by IoT contain a tremendous amount of information for anomaly detection. The heterogeneous nature of IoT is both a challenge and an opportunity for cybersecurity. Traditional approaches in cybersecurity monitoring often require different kinds of data pre-processing and handling for various data types, which might be problematic for datasets that contain heterogeneous features. However, heterogeneous types of network devices can often capture a more diverse set of signals than a single type of device readings, which is particularly useful for anomaly detection. In this paper, we present a comprehensive study on using ensemble machine learning methods for enhancing IoT cybersecurity via anomaly detection. Rather than using one single machine learning model, ensemble learning combines the predictive power from multiple models, enhancing their predictive accuracy in heterogeneous datasets rather than using one single machine learning model. We propose a unified framework with ensemble learning that utilises Bayesian hyperparameter optimisation to adapt to a network environment that contains multiple IoT sensor readings. Experimentally, we illustrate their high predictive power when compared to traditional methods.

Machine learning approaches have exhibited high potential in power system transient stability assessment (TSA), yet their initial preparation stages of stability knowledge base generation (SKBG) based on time-domain simulations often undergo high computational costs. In fact, how to ease the heavy computational burden of SKBG without sacrificing the reliability is still a significant challenge. To address this problem, this paper develops a semi-supervised ensemble learning (SSEL) framework for reliable SKBG acceleration, without additional need for computing hardware upgrading. In particular, it performs detailed simulations for a minority of cases and fast simulations for the majority ones to reduce the total computation time. Considering the absence of stability status information of those fast simulated cases, an SSEL scheme is systematically designed to reliably label their stability status. Before implementing SSEL, two concise feature descriptors are first introduced to efficiently extract transient features from multiplex system trajectories. Then, all the cases are characterized in a unified feature space, whereby a series of semi-supervised support vector machines are trained in randomly formed subspaces. Afterward, these single machines are systematically combined to derive an enhanced SSEL model, which is able to make reliable and robust labeling decisions. Further, a backtrace strategy is carefully devised for SSEL, so as to maintain the high reliability of SKBG. Test results on the IEEE 39-bus system and the realistic GD Power Grid in South China illustrate the excellent performances of the proposed framework on SKBG acceleration.

Predicting Students at Risk of Early Dropping Out from Course Using Ensemble Classification Methods

<p>Ensemble learning is one of the most powerful extensions for improving upon individual machine learning models. Rather than a single model being used, several models are trained and the predictions combined to make a more informed decision. Such combinations will ideally overcome the shortcomings of any individual member of the ensemble. Most ma- chine learning competition winners feature an ensemble of some sort, and there is also sound theoretical proof to the performance of certain ensem- bling schemes. The benefits of ensembling are clear in both theory and practice. Despite the great performance, ensemble learning is not a trivial task. One of the main difficulties is designing appropriate ensembles. For exam- ple, how large should an ensemble be? What members should be included in an ensemble? How should these members be weighted? Our first contribution addresses these concerns using a strongly-typed population- based search (genetic programming) to construct well-performing ensem- bles, where the entire ensemble (members, hyperparameters, structure) is automatically learnt. The proposed method was found, in general, to be significantly better than all base members and commonly used compari- son methods trialled. With automatically designed ensembles, there is a range of applica- tions, such as competition entries, forecasting and state-of-the-art predic- tions. However, often these applications also require additional prepro- cessing of the input data. Above the ensemble considers only the original training data, however, in many machine learning scenarios a pipeline is required (for example performing feature selection before classification). For the second contribution, a novel automated machine learning method is proposed based on ensemble learning. This method uses a random population-based search of appropriate tree structures, and as such is em- barrassingly parallel, an important consideration for automated machine learning. The proposed method is able to achieve equivalent or improved results over the current state-of-the-art methods and does so in a fraction of the time (six times as fast). Finally, while complex ensembles offer great performance, one large limitation is the interpretability of such ensembles. For example, why does a forest of 500 trees predict a particular class for a given instance? In an effort to explain the behaviour of complex models (such as ensem- bles), several methods have been proposed. However, these approaches tend to suffer at least one of the following limitations: overly complex in the representation, local in their application, limited to particular fea- ture types (i.e. categorical only), or limited to particular algorithms. For our third contribution, a novel model agnostic method for interpreting complex black-box machine learning models is proposed. The method is based on strongly-typed genetic programming and overcomes the afore- mentioned limitations. Multi-objective optimisation is used to generate a Pareto frontier of simple and explainable models which approximate the behaviour of much more complex methods. We found the resulting rep- resentations are far simpler than existing approaches (an important con- sideration for interpretability) while providing equivalent reconstruction performance. Overall, this thesis addresses two of the major limitations of existing ensemble learning, i.e. the complex construction process and the black- box models that are often difficult to interpret. A novel application of ensemble learning in the field of automated machine learning is also pro- posed. All three methods have shown at least equivalent or improved performance than existing methods.</p>

Reviewing ensemble classification methods in breast cancer

This chapter introduces ensemble learning and provides an overview of ensemble methods for class imbalance learning (CIL). Ensemble methods use a set of classifiers to make predictions. The generalization ability of an ensemble is usually much stronger than that of the individual ensemble members. In CIL, ensemble methods are broadly used to further improve the existing methods or help design brand new ones. According to how the base learners are generated, ensemble methods can be roughly categorized into two paradigms: parallel ensemble methods and sequential ensemble methods. According to what ensemble method is involved, ensemble methods for CIL can be roughly categorized into Bagging-style methods, boosting-based methods, hybrid ensemble methods, and other methods. Most of the ensemble methods for CIL deal with binary class problems. How to use ensemble learning to help multiclass problems is an interesting direction.