Multiobjective genetic classifier selection for random oracles fuzzy rule-based classifier ensembles: How beneficial is the additional diversity?

Diversity is a key characteristic of a classifier ensemble. A classifier ensemble must be composed of base classifiers with different performance in different areas of the problem space. Several works studied different diversity measures by performing extensive numerical experiments. However, up to our knowledge, no method has been proposed to visualize the diversity of a classifier ensemble. In this contribution, we propose a novel approach to visualize and analyze classifier ensembles diversity. Our novel proposal (called DivGrams), allows the visualization of the diversity between base classifiers in the classifier ensemble by using a network representation. The nodes represent base classifiers and its edges represent diversity relations between the base classifiers. We apply DivGrams to fuzzy rule-based classifier ensembles, as in our previous work these machine learning models proved to deal well with complex classification problems. A preliminary experimental study is performed using two UCI datasets in order to show the usefulness of DivGrams. Furthermore, we show a practical application of DivGrams to develop manual classifier selection on eight UCI datasets. The results obtained show the interesting characteristics of our proposal and support this novel approach in the fuzzy classifier ensemble field.

Subcellular localization is a key functional characteristic of proteins. An automatic, reliable and efficient prediction system for protein subcellular localization can be used for establishing knowledge of the spatial distribution of proteins within living cells and permits to screen systems for drug discovery or for early diagnosis of a disease. In this paper, we investigate an approach based on augmented image features by incorporating curvelet transform and neural network (MLP) ensemble for classification. A simple Random Subspace (RS) ensemble offers satisfactory performance, which contains a set of base MLP classifiers trained with subsets of attributes randomly drawn from the combined features of curvelet coefficients and original Subcellular Location Features (SLF). An MLP ensemble with Random Linear Oracle (RLO) can further improve the performance by replacing a base classifier with a miniensemble, which consists of a pair of base classifiers and a fixed, randomly created oracle that selects between them. With the benchmarking 2D HeLa images, our experiments show the effectiveness of the proposed approach. The RS-MLP ensemble offers the classification rate 95% while the RS-RLO ensemble gives 95.7% accuracy, which compares sharply with the previously published benchmarking result 84%.

In the Random Oracle ensemble method, each base classifier is a mini-ensemble of two classifiers and a randomly generated oracle that selects one of the two classifiers. The performance of this method have been previously studied, but not for imbalanced data sets. This work studies its performance for this kind of data. As the Random Oracle ensemble method can be combined with any other ensemble method, this work considers its combination with four ensemble methods: Bagging, SMOTEBoost, SMOTEBagging and RUSBoost. The last three methods combine classical, not specific for imbalance, ensemble methods (i.e., Bagging, Boosting), with pre-processing approaches designed for imbalance (i.e., random undersampling, SMOTE). The results show that Random Oracles improves all these methods.

In the last decades, the use of ensembles of classifiers in different domains of application has received significant attention of the Machine Learning community. In a typical architecture of ensemble, a new input pattern is presented to all K components. The individual classifiers provide their output and send them to a combination method, which is responsible for providing the final output. Ensemble of classifiers can be applied to any classification problem, single-label or multi- label problems. Multi-label (ML) classification has been received much attention from various research domains, such as text categorization, bioinformatics, computer vision, among others. In this paper, we investigate the use of ensemble of classifiers when applied to multi-label problems, focusing on the use of diversity measures in these systems when applied to multi-label problems. In order to do this, the present work proposes an adapted approach for two well-known diversity measures, good and bad, for ensembles of multi-label classifiers. In order to assess the feasibility of the proposed diversity measures in ensemble of multi-label classifiers, an empirical analysis is conducted, with eight multi-label classification problems. Generally, our finds indicate that there is a strong correlation between performance of the ML ensembles with the proposed diversity measures, showing that these diversity measures can be used as an important tool in the design of efficient ensembles of multi-label classifiers.

The traditional hierarchical identity fully homomorphic encryption scheme can only satisfy the homomorphic operation under a single identity, and the dimension of the lattice will increase synchronously with the increase of the layering depth. Aiming at the problem of high algorithm complexity due to the high-precision real number iterative operation used in the trapdoor derivation process of the existing scheme, this paper proposes an improved multi-identity fully homomorphic encryption scheme (mIBFHE): first, we use the features of new MP12 trapdoor to optimize the Gaussian sampling of the original scheme; secondly, the encryption and decryption structure of the original scheme is modified on the basis of the dual Regev system, so that the improved hierarchical identity encryption (HIBE) scheme meets the conversion format of the GSW compiler; finally, the transformed HIBFHE scheme is further transformed into a multi-identity fully homomorphic encryption scheme through the Mask system that meets the conditions. The hardness of our scheme can be reduced to the LWE problem, and IND-ID-CPA security is achieved under the random oracle model. After comparative analysis, the size of the master private key of the mIBFHE scheme in this paper is significantly reduced compared to the size of the same type of IBE scheme, which is 1/4 of the original scheme size. Our scheme is more efficient than traditional IBFHE scheme that it has smaller size than similar schemes in both the public key and the ciphertext when satisfying the attributes of hierarchical, homomorphism, and multiple identities and at the same time.

Achieving high classification accuracy is vital in reliable information extraction from images. Single classifiers and existing ensemble methods suffer from data dimensionality, insufficient ground truth information and lack in defining optimal feature selection. This article presents a novel idea for constructing component classifiers that boost random subspace ensemble method in improving its classification performance. It is achieved through sub-optimal training of component classifiers through interference in training process during validation error evaluation. The new approach allows to enforce different class errors among component classifiers, besides improving individual class accuracy. This article demonstrates effectiveness of the anti-cross validation approach using three classical hyperspectral Image (HSI) datasets with significant improvement in classification accuracies from 3 to 10% with the proposed approach.

Ensembles of extreme learning machine (ELM) have been widely used for hyperspectral image classification. The previous studies have shown that the goal of ensemble learning is to train accurate but diverse component classifiers to improve the generalization performance. To approach this goal, this article proposes a novel framework to construct an ELM ensemble model. The proposed framework relies on multiview-based random rotation ensemble pruning (MVRR-EP) and has several novel features. First, to ensure that the subsets of spectral bands can sufficiently learn the target concept, the spectral bands are divided into multiviews by using correlation analysis. Second, random rotation, a new approach of space transformation, is introduced to transform each view into multiple coordinate spaces, which makes the component classifiers trained on the transformed spaces have great diversity. Third, an accuracy guided ensemble pruning strategy is designed for pruning the component classifiers with low complementarity, and consequently, the remaining component classifiers with high complementarity are combined to a construct ensemble classifier. These techniques guarantee that the component classifiers used to construct an ensemble classifier are accurate but diverse, which ultimately improves the performance of the ensemble classifier. To demonstrate the effectiveness of the proposed MVRR-EP, extensive experiments were carried out on four hyperspectral data sets. Experimental results verify that compared with other methods, the proposed method provides competitive results.

This chapter describes several new fields, beside the feature selection preprocessing step (the theme of this book), in which ensembles have been successfully used. First, in Sect. 7.1, we introduce a very brief review of the different application fields in which ensembles have been applied, together with basic levels that are used to produce different ensemble designs, and a sample taxonomy. Then, in Sect. 7.2 basic ideas in ensemble classification design, one of the very first machine learning areas in which the idea of ensembles was applied, are stated. As there are many interesting and reference books in ensemble classification, we focus on describing the latest ideas in classification ensembles that address problems such as classification, stream data, missing data and imbalance data. Afterwards, the first attempts for applying the ensemble paradigm to the relatively new field of quantification are described in Sect. 7.3. In Sect. 7.4 we move to describing ensembles for clustering, another area in which ensembles have been increasingly popular. In Sect. 7.5 an attempt on ensembles for discretization is described and, finally, Sect. 7.6 summarizes and discusses the contents of this chapter.

Random Linear Oracle (RLO) ensembles of Naive Bayes classifiers show excellent performance [12]. In this paper, we investigate the reasons for the success of RLO ensembles. Our study suggests that the decomposition of most of the classes of the dataset into two subclasses for each class is the reason for the success of the RLO method. Our study leads to the development of a new output manipulation based ensemble method; Random Subclasses (RS). In the proposed method, we create new subclasses from each subset of data points that belongs to the same class using RLO framework and consider each subclass as a class of its own. The comparative study suggests that RS is similar to RLO method, whereas RS is statistically better than or similar to Bagging and AdaBoost.M1 for most of the datasets. The similar performance of RLO and RS suggest that the creation of local structures (subclasses) is the main reason for the success of RLO. The another conclusion of this study is that RLO is more useful for classifiers (linear classifiers etc.) that have limited flexibility in their class boundaries. These classifiers can not learn complex class boundaries. Creating subclasses makes new, easier to learn, class boundaries.

Rough set theory has proved efficient for many applications, including finding hidden patterns in data, data reduction, evaluating significance of data, and generating sets of decision rules from data. Recently, it has shown to be effective approach for constructing ensemble learning systems as well. Learning classifier systems are genetics-based machine learning techniques that have recently shown a high degree of competence on a variety of data mining problems. Attempts to improve its generalization capabilities in the literature using ensemble learning lack a systematic and robust techniques for partitioning the problem at hand. It is well known that ensemble performance depends on the problem decomposition technique being used. Rough set based learning classifier system ensemble is proposed in this paper. In this approach, rough set attribute reduction is used to generate a set of reducts, and then a diverse subset of these reducts is selected to train an ensemble of base classifiers. The experiments show that classification accuracy of reduct-based ensemble systems outperforms a single learning classifier system model. It has also shown better performance than either an ensemble of classifiers with all attributes being used or a single classifier trained by a single reduct. It has also shown competitive performance to the random subspace ensemble strategy on the set of real data sets used in the experiments.

Impact of the learners diversity and combination method on the generation of heterogeneous classifier ensembles

A computer aided diagnosis system aiming to classify liver tissue from computed tomography images is presented. For each region of interest five distinct sets of texture features were extracted. Two different ensembles of classifiers were constructed and compared. The first one consists of five Neural Networks (NNs), each using as input either one of the computed texture feature sets or its reduced version after feature selection. The second ensemble of classifiers was generated by combining five different type of primary classifiers, two NNs, and three k-nearest neighbor classifiers. The primary classifiers of the second ensemble used identical input vectors, which resulted from the combination of the five texture feature sets, either directly or after proper feature selection. The decision of each ensemble of classifiers was extracted by applying voting schemes. © 2006 International Federation for Information Processing.

Bio-inspired and gradient-based algorithms to train MLPs: The influence of diversity

An integrated TOPSIS crow search based classifier ensemble: In application to stock index price movement prediction

K -nearest neighbor classification method (KNN), as one of the top 10 algorithms in data mining, is a very simple and yet effective nonparametric technique for pattern recognition. However, due to the selective sensitiveness of the neighborhood size k , the simple majority vote, and the conventional metric measure, the KNN-based classification performance can be easily degraded, especially in the small training sample size cases. In this article, to further improve the classification performance and overcome the main issues in the KNN-based classification, we propose a local mean representation-based k -nearest neighbor classifier (LMRKNN). In the LMRKNN, the categorical k -nearest neighbors of a query sample are first chosen to calculate the corresponding categorical k -local mean vectors, and then the query sample is represented by the linear combination of the categorical k -local mean vectors; finally, the class-specific representation-based distances between the query sample and the categorical k -local mean vectors are adopted to determine the class of the query sample. Extensive experiments on many UCI and KEEL datasets and three popular face databases are carried out by comparing LMRKNN to the state-of-art KNN-based methods. The experimental results demonstrate that the proposed LMRKNN outperforms the related competitive KNN-based methods with more robustness and effectiveness.