A network-based approach for diversity visualization of fuzzy classifier ensembles

In the last decades, the use of ensembles of classifiers in different domains of application has received significant attention of the Machine Learning community. In a typical architecture of ensemble, a new input pattern is presented to all K components. The individual classifiers provide their output and send them to a combination method, which is responsible for providing the final output. Ensemble of classifiers can be applied to any classification problem, single-label or multi- label problems. Multi-label (ML) classification has been received much attention from various research domains, such as text categorization, bioinformatics, computer vision, among others. In this paper, we investigate the use of ensemble of classifiers when applied to multi-label problems, focusing on the use of diversity measures in these systems when applied to multi-label problems. In order to do this, the present work proposes an adapted approach for two well-known diversity measures, good and bad, for ensembles of multi-label classifiers. In order to assess the feasibility of the proposed diversity measures in ensemble of multi-label classifiers, an empirical analysis is conducted, with eight multi-label classification problems. Generally, our finds indicate that there is a strong correlation between performance of the ML ensembles with the proposed diversity measures, showing that these diversity measures can be used as an important tool in the design of efficient ensembles of multi-label classifiers.

Effectively utilizing multimodal information (e.g., heart rate and acceleration) is a promising way to achieve wearable sensor based human activity recognition (HAR). In this paper, an activity recognition approach MARCEL (Multimodal Activity Recognition with Classifier Ensemble) is proposed, which exploits the diversity of base classifiers to construct a good ensemble for multimodal HAR, and the diversity measure is obtained from both labeled and unlabeled data. MARCEL uses neural network (NN) as base classifiers to construct the HAR model, and the diversity of classifier ensemble is embedded in the error function of the model. In each iteration, the error of the model is decomposed and back-propagated to base classifiers. To ensure the overall accuracy of the model, the weights of base classifiers are learnt in the classifier fusion process with sparse group lasso. Extensive experiments show that MARCEL is able to yield a competitive HAR performance, and has its superiority on exploiting multimodal signals.

Multiple Classifier System (MCS) has attracted increasing interest in the field of pattern recognition and machine learning where this technique has also been introduced in remote sensing. The importance of classifier diversity in MCS has been raised recently; nevertheless, only a few of the researches have been studied in land cover classification problem. In this paper, a SPOT IV satellite image covering the Hong Kong Island and Kowloon Peninsula with six land cover classes were classified with four base classifiers: Minimum Distance Classifier, Maximum Likelihood Classifier, Mahalanobis Classifier and K-Nearest Neighbor Classifier. Same training and testing data sets were applied throughout the experiments and five Bayesian decision rules, including product rule, sum rule, max rule, min rule, and median rule, were utilized to construct different ensemble of classifiers. Performance of MCS was measured using the overall accuracy and kappa statistics, and three statistical tests including McNemarpsilas test, Cochranpsilas Q test and F-test were introduced to examine the dependence of the classification results. The experimental comparison reveals that (i) increasing the number of base classifiers may not improve the overall accuracy in MCS, (ii) significant diversity in base classifiers cannot enhance the overall performance and vice versa. These findings are noted with the condition in using the same data set and the same training set.

Non-parametric classification procedures based on a certainty measure and nearest neighbour rule for motor unit potential classification (MUP) during electromyographic (EMG) signal decomposition were explored. A diversity-based classifier fusion approach is developed and evaluated to achieve improved classification performance. The developed system allows the construction of a set of non-parametric base classifiers and then automatically chooses, from the pool of base classifiers, subsets of classifiers to form candidate classifier ensembles. The system selects the classifier ensemble members by exploiting a diversity measure for selecting classifier teams. The kappa statistic is used as the diversity measure to estimate the level of agreement between base classifier outputs, i.e., to measure the degree of decision similarity between base classifiers. The pool of base classifiers consists of two kinds of classifiers: adaptive certainty-based classifiers (ACCs) and adaptive fuzzy k-NN classifiers (AFNNCs) and both utilize different types of features. Once the patterns are assigned to their classes, by the classifier fusion system, firing pattern consistency statistics for each class are calculated to detect classification errors in an adaptive fashion. Performance of the developed system was evaluated using real and simulated EMG signals and was compared with the performance of the constituent base classifiers and the performance of the fixed ensemble containing the full set of base classifiers. Across the EMG signal data sets used, the diversity-based classifier fusion approach had better average classification performance overall, especially in terms of reducing classification errors.

To predict potential problems and avoid an unexpected breakdown of rotating machinery, a rule-based classifier ensemble approach is presented. Feature reduction was first implemented on a fault decision table using discernibility matrices and the genetic algorithm. The generated rules of the reducts were used to build the candidate base classifiers. Then, several base classifiers were selected according to their diversity and scale. The weights of the selected base classifiers were also calculated based on the support rate measurements. A classifier ensemble was constructed through an integration of the base classifiers using an improved weighted voting technique. Finally, the proposed classifier ensemble was verified based on the vibration data of bearing types SKF6203 and NU205. The accuracy for the SKF6203 bearing type reached 88.75 %, which is at least 5 % higher than that of the three base classifiers for this type of bearing. In addition, the recognition rate for the latter bearing type was 90 %. The reasoning process was much easier to comprehend owing to the semantic descriptions of the rules. The results show that this is a promising and transparent approach for diagnosing typical faults of rotating machinery.

Ensembles need their base classifiers do not always agree for any prediction (diverse base classifiers). Disturbing Neighbors ($\mathcal{DN}$) is a method for improving the diversity of the base classifiers of any ensemble algorithm. $\mathcal{DN}$ builds for each base classifier a set of extra features based on a 1-Nearest Neighbors (1-NN) output. These 1-NN are built using a small subset of randomly selected instances from the training dataset. $\mathcal{DN}$ has already been proved successfully on unstable base classifiers (i.e. decision trees). This paper presents an experimental validation on 62 UCI datasets for standard ensemble methods using Support Vector Machines (SVM) with a linear kernel as base classifiers. SVMs are very stable, so it is hard to increase their diversity when they belong to an ensemble. However, experiments will show that $\mathcal{DN}$ usually improves ensemble accuracy and base classifiers diversity.

Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms

One of the major developments in machine learning in the past decade is the ensemble method, which finds highly accurate classifier by combining many moderately accurate component classifiers. In this research work, new ensemble classification methods are proposed with homogeneous ensemble classifier using bagging and heterogeneous ensemble classifier using arcing and their performances are analyzed in terms of accuracy. A Classifier ensemble is designed using Radial Basis Function RBF and Support Vector Machine SVM as base classifiers. The feasibility and the benefits of the proposed approaches are demonstrated by the means of standard datasets of direct marketing. The main originality of the proposed approach is based on three main parts: preprocessing phase, classification phase and combining phase. A wide range of comparative experiments are conducted for standard datasets of direct marketing. The performance of the proposed homogeneous and heterogeneous ensemble classifiers are compared to the performance of other standard homogeneous and heterogeneous ensemble methods. The standard homogeneous ensemble methods include Error correcting output codes, dagging and heterogeneous ensemble methods include majority voting, stacking. The proposed ensemble methods provide significant improvement of accuracy compared to individual classifiers and the proposed bagged RBF and SVM performs significantly better than ECOC and Dagging and the proposed hybrid RBF-SVM performs significantly better than voting and stacking. Also heterogeneous models exhibit better results than homogeneous models for standard datasets of direct marketing.

Proposing a classifier ensemble framework based on classifier selection and decision tree

An integrated TOPSIS crow search based classifier ensemble: In application to stock index price movement prediction

This paper proposes a reduct construction method based on discernibility matrix simplification. The method works with genetic algorithm. To identify potential problems and prevent complete failure of bearings, a new method based on rule-based classifier ensemble is presented. Genetic algorithm is used for feature reduction. The generated rules of the reducts are used to build the candidate base classifiers. Then, several base classifiers are selected according to their diversity and the scale of them. Weights of the selected base classifiers are calculated based on a measure of support rate. The classifier ensemble is constructed by the base classifiers. The accuracy reached 98.44% which is 4.5% higher than that of the three base classifiers.

With the rising demand for integrated and autonomous systems in the field of engineering, efficient frameworks for instant detection of performance anomalies are imperative for improved productivity and cost-effectiveness. This study proposes a systematic predictive maintenance framework based on the hybrid multisensor fusion technique of fuzzy rough set feature selection and stacked ensemble for the efficient classification of fault conditions characterised by uncertainties. First, a feature vector of time-domain features was extracted from 17 multiple sensor signals. Then, a comparative study of six different Fuzzy Rough Set Feature Selection (FRFS) methods was employed to select the various combinations of optimal feature subsets for various faults classification tasks. The determined optimal feature subsets then served as inputs for training the stacked ensemble (ESB(STK)). In the ESB(STK), Support Vector Machine (SVM), Multilayer Perceptron (MLP), k -Nearest Neighbour ( k -NN), C4.5 Decision Tree (C4.5 DT), Logistic Regression (LR), and Linear Discriminant Analysis (LDA) served as the base classifiers while the LR was selected to be the metaclassifier. The proposed hybrid framework (FRFS-ESB(STK)) improved the classification accuracy with the selected combinations of optimal feature subset size whiles reducing the computational cost, overfitting, training runtime, and uncertainty in modelling. Overall analyses showed that the FRFS-ESB(STK) proved to be generalisable and versatile in the classification of all conditions of four monitored hydraulic components (i.e., cooler, valve, accumulator, and internal pump leakage) when compared with the six base classifiers (standalone) and three existing ensemble classifiers (Stochastic Gradient Boosting (SGB), Ada Boost (ADB), and Bagging (BAG)). The proposed FRFS-ESB(STK) showed an average improvement of 11.28% and 0.88% test accuracies when classifying accumulator and pump conditions, respectively, whiles 100% classification rates were obtained for both cooler and valve.

This paper introduces a novel concept for creating an ensemble of classifiers. The concept is based on generating an ensemble of classifiers through clustering of data at multiple layers. The ensemble classifier model generates a set of alternative clustering of a dataset at different layers by randomly initializing the clustering parameters and trains a set of base classifiers on the patterns at different clusters in different layers. A test pattern is classified by first finding the appropriate cluster at each layer and then using the corresponding base classifier. The decisions obtained at different layers are fused into a final verdict using majority voting. As the base classifiers are trained on overlapping patterns at different layers, the proposed approach achieves diversity among the individual classifiers. Identification of difficult-to-classify patterns through clustering as well as achievement of diversity through layering leads to better classification results as evidenced from the experimental results.

Feature selection and ensemble classification increase system efficiency and accuracy in machine learning, data mining and biomedical informatics. This research presents an analysis of the effect of removing irrelevant and redundant features with ensemble classifiers using two datasets from UCI machine learning repository. Accuracy and computational time were evaluated by four base classifiers; NaiveBayes, Multilayer Perceptron, Support Vector Machines and Decision Tree. Eliminating irrelevant features improves accuracy and reduces computational time while removing redundant features reduces computational time and reduces accuracy of the ensemble.

Network intrusion behaviour data is the imbalanced data. It includes a large amount of normal behavior data and a small amount of intrusion behavior data. The traditional selective ensemble learning algorithm will lead to high false negative rate. This paper proposes a selective ensemble learning algorithm for imbalanced data based on under sampling (SELAUS). First of all, the algorithm uses Bootstrap method to extract samples equal to the number of samples of a few classes from majority class samples to construct multiple balanced training subsets. Then, in order to ensure that the obtained base classifiers have great differences, several features are randomly selected on the training subset and a decision tree is constructed as the base classifier using CART algorithm. This method can also make some base classifiers have poor performance, so it can select and integrate base classifiers instead of all base classifiers. In order to accurately evaluate the generalization error of the classifier for imbalanced dataset, this paper defines the performance evaluation method in the imbalanced dataset and the difference evaluation method between the base classifiers. Then the generalization error of each base classifier is calculated, and the base classifier is selected according to the generalization error. In the integration of weighted voting, the weight of each base classifier is calculated by the weight calculation method for imbalanced data. Finally, the validity of the algorithm is verified by UCI dataset and applied to network intrusion detection. The simulation results show that the algorithm can improve the detection rate of minority class samples, that is to say, reduce the false negative rate.