Building Weighted Classifier Ensembles Through Classifiers Pruning

Cancer is a group of related diseases with high mortality rate characterized by abnormal cell growth which attacks the body tissues. Microarray cancer data is a prominent research topic across many disciplines focused to address problems related to the higher curse of dimensionality, a small number of samples, noisy data and imbalance class. A random forest (RF) tree‐based feature selection and ensemble learning based on hard voting and soft voting is proposed to classify microarray cancer data using six different base classifiers. The selected features due to RF tree are submitted to the base classifiers as the training set. Then, an ensemble learning method is applied to the base classifiers in which case each base classifier predicts class label individually. The final prediction is carried out hard and soft voting techniques that use majority voting and weighted probability on the test set. The proposed ensemble learning method is validated on eight different standard microarray cancer datasets, of which three of the datasets are binary class and the remaining five datasets are multi‐class datasets. Experimental results of the proposed method show 1.00 classification accuracy on six of the datasets and 0.96 on two of the datasets.

The K Nearest Neighbor (kNN) method has widely been used in the applications of data mining and machine learning due to its simple implementation and distinguished performance. However, setting all test data with the same k value in the previous kNN methods has been proven to make these methods impractical in real applications. This article proposes to learn a correlation matrix to reconstruct test data points by training data to assign different k values to different test data points, referred to as the Correlation Matrix kNN (CM-kNN for short) classification. Specifically, the least-squares loss function is employed to minimize the reconstruction error to reconstruct each test data point by all training data points. Then, a graph Laplacian regularizer is advocated to preserve the local structure of the data in the reconstruction process. Moreover, an ℓ 1 -norm regularizer and an ℓ 2, 1 -norm regularizer are applied to learn different k values for different test data and to result in low sparsity to remove the redundant/noisy feature from the reconstruction process, respectively. Besides for classification tasks, the kNN methods (including our proposed CM-kNN method) are further utilized to regression and missing data imputation. We conducted sets of experiments for illustrating the efficiency, and experimental results showed that the proposed method was more accurate and efficient than existing kNN methods in data-mining applications, such as classification, regression, and missing data imputation.

An integrated TOPSIS crow search based classifier ensemble: In application to stock index price movement prediction

This paper introduces a novel concept for creating an ensemble of classifiers. The concept is based on generating an ensemble of classifiers through clustering of data at multiple layers. The ensemble classifier model generates a set of alternative clustering of a dataset at different layers by randomly initializing the clustering parameters and trains a set of base classifiers on the patterns at different clusters in different layers. A test pattern is classified by first finding the appropriate cluster at each layer and then using the corresponding base classifier. The decisions obtained at different layers are fused into a final verdict using majority voting. As the base classifiers are trained on overlapping patterns at different layers, the proposed approach achieves diversity among the individual classifiers. Identification of difficult-to-classify patterns through clustering as well as achievement of diversity through layering leads to better classification results as evidenced from the experimental results.

The speedy development of the amount of medical health information available to the healthcare industry is driving demand for more accurate, precise, and reliable tools for exploiting these data. Real-world experience suggests that combining the predictions of individual machine learning classifiers may allow us to meet all three criteria at a low cost, potentially saving scarce funds, and patient’s lives. We, therefore, attempt to go beyond the existing literature on individual machine learning algorithms to discover the effectiveness of combining their predictions, and to explore the potential benefits of such ensemble methods to personalized diagnostics and prognostic medicine. Consideration of a range of classifiers is an essential aspect of getting the best predictive performance for a given dataset. For each of the ten biomedical datasets considered in this paper, a range of classifier ensemble methods performed competitively concerning individual classifiers. Furthermore, it was possible to reliably automate the ensemble methods, making them straightforward to apply to new datasets. Strikingly, we found that merely selecting the best-performing individual model was a powerful strategy. This strategy outperforming most of the model ‘compounding’ approaches leading to best or near-best predictive performance on all ten datasets. We conclude that while it is essential to consider a range of classifiers for a given dataset, simple strategies such as selecting the best-performing base classifier should not be discounted. Also, there may be significant merit in the development of formal methods for selecting between a wide range of different classifiers.

Corrigendum to ‘Deep learning improves taphonomic resolution: high accuracy in differentiating tooth marks made by lions and jaguars'

Keyless entry systems have made human life more comfortable because the possibility of someone leaving a key is non-existent. However, there is a research opportunity to use Wi-Fi fingerprint localization in an indoor keyless entry system. The use of Wi-Fi as a solution is low cost because of the already hdeployed Wi-Fi access points (AP) around buildings. This study uses ensemble learning to utilize Wi-Fi fingerprints for keyless entry systems in indoor environments. The first step of this research is to obtain the UJIIndoorLoc Data Set, an open-source Wi-Fi fingerprint dataset. The next step is to design and implement a keyless entry system based on Wi-Fi Fingerprint. We compared four ensemble learning methods: Random Forest, AdaBoost, gradient boosting, and XGBoost. We use several metrics to compare the four methods, namely r2, the area under the receiver operating curve (AUROC), and the equal error rate (EER). The test results show that for localization based on ensemble regression, XGBoost has the best performance compared to the other three ensemble methods for both longitude and latitude predictions. The r2values are 0.94 and 0.98, respectively. For authentication based on ensemble classification, the gradient-boosting model performance increases as the number of weak learners increases. The optimum number of weak learners is 60. With AUROC = 0.99, gradient boosting outperforms random forest, AdaBoost, and XGBoost on authentication. Finally, our authentication method has EER = 0.01, which outperforms other state-of-the-art keyless entry systems. This research focuses on local building map datasets for future work.

One of the major developments in machine learning in the past decade is the ensemble method, which finds highly accurate classifier by combining many moderately accurate component classifiers. In this research work, new ensemble classification methods are proposed with homogeneous ensemble classifier using bagging and heterogeneous ensemble classifier using arcing and their performances are analyzed in terms of accuracy. A Classifier ensemble is designed using Radial Basis Function RBF and Support Vector Machine SVM as base classifiers. The feasibility and the benefits of the proposed approaches are demonstrated by the means of standard datasets of direct marketing. The main originality of the proposed approach is based on three main parts: preprocessing phase, classification phase and combining phase. A wide range of comparative experiments are conducted for standard datasets of direct marketing. The performance of the proposed homogeneous and heterogeneous ensemble classifiers are compared to the performance of other standard homogeneous and heterogeneous ensemble methods. The standard homogeneous ensemble methods include Error correcting output codes, dagging and heterogeneous ensemble methods include majority voting, stacking. The proposed ensemble methods provide significant improvement of accuracy compared to individual classifiers and the proposed bagged RBF and SVM performs significantly better than ECOC and Dagging and the proposed hybrid RBF-SVM performs significantly better than voting and stacking. Also heterogeneous models exhibit better results than homogeneous models for standard datasets of direct marketing.

Recently, various ensemble learning methods with different base classifiers have been proposed for credit scoring problems. However, for various reasons, there has been little research using logistic regression as the base classifier. In this paper, given large unbalanced data, we consider the plausibility of ensemble learning using regularized logistic regression as the base classifier to deal with credit scoring problems. In this research, the data is first balanced and diversified by clustering and bagging algorithms. Then we apply a Lasso-logistic regression learning ensemble to evaluate the credit risks. We show that the proposed algorithm outperforms popular credit scoring models such as decision tree, Lasso-logistic regression and random forests in terms of AUC and F-measure. We also provide two importance measures for the proposed model to identify important variables in the data.

Polymer-electrolyte aqueous two-phase systems (ATPS) have demonstrated their superior performance in the separation and purification of high-value biomolecules. However, these powerful platforms are still a major academic curiosity, without their acceptance and implementation by industry. One of the major obstacles is the absence of models to predict the partition of biomolecules in ATPS in an easy and predictive way. To address this limitation, modelling studies on the binodal curve behavior of polymer-electrolyte ATPS and the partitioning of biomolecules in these aqueous electrolyte solutions are carried out in this work. First, a comprehensive database targeting the studied systems is established. In total, 11,998 experimental binodal data points covering 276 polymer-electrolyte ATPS at different temperatures (273.15 K-399.15 K) and 626 experimental partition data points involving 22 biomolecules in 42 polymer-electrolyte ATPS at different temperatures (283.15 K-333.15 K) are included. Then, a novel modeling strategy that combines a well-known machine learning algorithm, i.e., artificial neural network (ANN) and group contribution (GC) method is proposed. Based on this modeling strategy, an ANN-GC model (ANN-GC model1) is built to describe the binodal curve behavior of polymer-electrolyte ATPS, while another ANN-GC model (ANN-GC model2) is developed to predict the partition of biomolecules in these biphasic systems. ANN-GC model1 gives a mean absolute error (MAE) of 0.0132 and squared correlation coefficient (R2) of 0.9878 for the 9,598 training data points, and for the 1,200 validation data points they are 0.0141 and 0.9858, respectively. Meanwhile, it also gives a MAE of 0.0143 and R2 of 0.9846 for the 1,200 test data points. On the other hand, ANN-GC model2 gives root-mean-square deviation (RMSD) of 0.0577 for 501 training data points, and for the 62 validation data points and 63 test data points their RMSD are 0.0849 and 0.0885, respectively. Furthermore, the obtained results also indicate that the tie-line length of polymer-electrolyte ATPS calculated from ANN-GC model1 can be directly used in ANN-GC model2 for predicting the partition performance coefficient of biomolecules in these ATPS. The developed models offer the possibility to predict the partition of biomolecules in ATPS without any requirement of experimental data. Based on the developed ANN-GC models, some high-performance ATPS are identified to partition four well-known biomolecules.

Ensemble Learning is a popular learning paradigm and finds its application in many diverse fields. Random Forest, a decision tree based ensemble learning algorithm has received constant attention in the research community due to its ability to learn complex rules and generalize well for unknown data. Identifying the number of base classifiers (trees) required for a particular dataset is one of the key questions addressed in this paper. Statistical analyses of individual base classifiers are carried out to prune the ensemble model without compromising the classification accuracy of the model. Learning the learned model, i.e., learning the statistics of the forest in its entirety along with the information available in the dataset can reveal the optimal thresholds that should be used to prune an ensemble model. Experimental results reveal that, on an average 78% of the trees were pruned on 26 different datasets obtained from the UCI repository. The impact of pruning was positive with 22 out of 26 datasets showing equal or better classification accuracy in comparison with the Classical Random Forest algorithm.

Perfume identification (PI) based on an electronic nose (EN) can be used for exposing counterfeit perfumes more time-efficiently and cost-effectively than using gas chromatography and mass spectrometry instruments. During the past five years, decision-tree-based ensemble learning methods, also called tree ensemble learning methods, have demonstrated excellent performance when solving multi-class classification problems. However, the performance of tree ensemble learning methods for the EN-based PI problem remains uncertain. In this paper, four well-known tree ensemble learning classification methods, random forest (RF), stagewise additive modeling using a multi-class exponential loss function (SAMME), gradient-boosting decision tree (GBDT), and extreme gradient boosting (XGBoost), were implemented for PI using our self-designed EN. For fair comparison, all the tested classification methods used as input the same feature data extracted using principal component analysis. Moreover, two benchmark methods, neural network and support vector machine, were also tested with the same experimental setup. The quantitative results of experiments undertaken demonstrated that the mean PI accuracy achieved by XGBoost was up to 97.5%, and that XGBoost outperformed other tested methods in terms of accuracy mean and variance based on our self-designed EN.

Detecting harmful content or hate speech on social media is a significant challenge due to the high throughput and large volume of content production on these platforms. Identifying hate speech in a timely manner is crucial in preventing its dissemination. We propose a novel stacked ensemble approach for detecting hate speech in English tweets. The proposed architecture employs an ensemble of three classifiers, namely support vector machine (SVM), logistic regression (LR), and XGBoost classifier (XGB), trained using word2vec and universal encoding features. The meta classifier, LR, combines the outputs of the three base classifiers and the features employed by the base classifiers to produce the final output. It is shown that the proposed architecture improves the performance of the widely used single classifiers as well as the standard stacking and classifier ensemble using majority voting. We also present results on the use of various combinations of machine learning classifiers as base classifiers. The experimental results from the proposed architecture indicated an improvement in the performance on all four datasets compared with the standard stacking, base classifiers, and majority voting. Furthermore, on three of these datasets, the proposed architecture outperformed all state-of-the-art systems.

Multi-model ensemble learning for battery state-of-health estimation: Recent advances and perspectives

Sentiment analysis extracts the emotions expressed in text and has been employed in many fields including politics, elections, movies, retail businesses and in recent years microblogs to understand, track and control the human sentiments or reactions toward products events or ideas. Nevertheless challenges such as different styles of writing, use of negation and sarcasm, existence of spelling mistakes, invention of new words etc. provide obstacle in the correct classification of sentiments. This paper provides an ensemble of classifiers framework for sentiment analysis. The proposed weighted majority voting ensemble method combines six models including Naïve Bayes, Logistic Regression, Stochastic Gradient Descent, Random Forest, Decision Tree and Support Vector Machine to form a single classifier. Weights of the individual classifiers of the ensemble are chosen as accuracy or Fl-score by optimizing their performance. This approach combines models based on the simple majority voting as opposed to the one based on weighted majority voting. Additionally, a comparison is drawn among these six individual classifiers to evaluate their performance. The proposed ensemble model is tested on some existing sentiment datasets, including SemEval 2017 Task 4A, 4B and 4C. The results demonstrate that the Logistic Regression classifier is optimal as compared to other individual classifiers. Furthermore, the proposed ensemble weighted majority voting classifier with the six individual classifiers performs better compared to the simple majority voting and all independent classifiers.