Comparative study of ensemble classifiers for direct marketing

The object of the study is the process of identifying anomalies in the operation of a computer system (CS). The subject of the study is ensemble methods for identifying the state of the CS. The goal of the study is to improve the performance of ensemble classifiers based on heterogeneous models. Methods used: machine learning methods, homogeneous and heterogeneous ensemble classifiers, Pasting and Bootstrapping technologies. Results obtained: a comparative analysis of the use of homogeneous and heterogeneous bagging ensembles in data classification problems was carried out. The effectiveness of various approaches to the selection of base ensemble classifiers has been studied. A method for identifying the state of a computer system, based on the heterogeneous bagging ensemble was proposed. Experimental studies made it possible to confirm the main theoretical assumptions, as well as evaluate the efficiency of the constructed heterogeneous ensembles. Conclusions. Based on the results of the study, the method for constructing a heterogeneous bagging ensemble classifier, which differs from known methods in the procedure for selecting base models was proposed. It made possible to increase the classification accuracy. Further development of this research could include the creating and integration of dissimilarity metrics as well as other quantitative metrics for a more accurate and balanced base model selection procedure, which would further improve the performance of the computer system state classifier.

In recent years, ensemble classification methods have been widely investigated in both industry and literature in the field of machine learning and artificial intelligence. The main advantage of this approach is to benefit from a set of classifiers instead of using a single classifier with the aim of improving the prediction performance, such as accuracy. Selecting the base classifiers and the method for combining them are the most challenging issues in the ensemble classifiers. In this paper, we propose a heterogeneous dynamic ensemble classifier (HDEC) which uses multiple classification algorithms. The main advantage of using heterogeneous algorithms is increasing the diversity among the base classifiers as it is a key point for an ensemble system to be successful. In this method, we first train many classifiers with the original data. Then, they are separated based on their strength in recognizing either positive or negative instances. For doing this, we consider the true positive rate and true negative rate, respectively. In the next step, the classifiers are categorized into two groups according to their efficiency in the mentioned measures. Finally, the outputs of the two groups are compared with each other to generate the final prediction. For evaluating the proposed approach, it has been applied to 12 datasets from the UCI and LIBSVM repositories and calculated two popular prediction performance metrics, including accuracy and geometric mean. The experimental results show the superiority of the proposed approach in comparison to other state-of-the-art methods.

Traditionally, the approach that has been used in the design of pattern classification was to experimentally assess the performance of several classifiers so that the best one would be chosen. Ensemble classifiers is developed to design a such reliable system with improved accuracy. Recognising that each classifier may make different or complementary errors, the idea is to pool the results from all classifiers to find a composite system that would out-perform any individual (base) classifier. In this way, a single complex classifier may be replaced by a set of relatively simple classifiers. Similarly, the idea of ensemble learning is to employ multiple learners and combine their predictions. Most ensemble methods use a single base learning algorithm to produce homogeneous base learners to produce homogeneous ensembles. Some methods use heterogeneous learners to create heterogeneous ensembles. For ensemble methods to be more accurate than any of its members, the base learners must be as accurate and as diverse as possible. Motivated by this ensemble approach, a heterogeneous ensemble model with different algorithms for members' training is proposed. More specifically, three types of classifiers namely the ELM, SVM and RF have been adopted for classification of Malay syllables. In the process of improving ensemble accuracy and stability, different combination methods are explored. These methods vary in their approach to treat the training data, the type of algorithms used, and the combination methods followed. Diversity among the performance of each single classifier in the ensemble classifiers is also analysed and compared with SD, SI, FI, FD, MI and MD configurations. Based on the obtained results, heterogeneous ensemble method has been successfully proven in improving recognition accuracy on the selected Malay syllables.

Background/Objectives: Recent studies emphasized on using ensemble models over single ones to solve credit scoring problems. The objective of this study is to build a heterogeneous ensemble classifier model with an improved classification accuracy. Methods: This study focuses on developing a heterogeneous ensemble classifier using Logistic Regression, K-nearest neighbor, Decision tree, Random Forest, Naïve Base and Support vector machine as base classifiers and Random Forest, Logistic Regression and Support vector machine as meta-classifiers. The proposed model is built using these six base classifiers for ensemble aggregation. A feature selection algorithm based on the random forest technique is used for selecting the best features. A stacking and voting method are used for building ensemble model. Findings: The ensemble classifier gives superior predictive performance than single classifiers SVM, DT, RF, NB, KNN and LR with an accuracy of 91.56% for Australian dataset and 84.35% for German dataset. Novelty: The proposed model uses stacking and majority voting method for ensemble classification. Initially, stacking is applied to the base classifiers. This is done in two levels. First the training dataset is split into 10 folds for cross validation. The output of each classifier is taken, and the dataset is updated with the meta-features. In the second level, three meta-classifiers (MC), namely LR, SVM and RF are used. Majority voting is applied to the output of these meta-classifiers for the prediction. Keywords: Credit scoring; ensemble model; SVM; DT; RF; NB; KNN; LR

Predictive models, which leverage the relationship between environmental variables and river health, serve as a valuable tool for predicting the river health at unmonitored sites. Such models should be generalizable to unseen data. However, predictions derived from machine learning (ML) models can exhibit large variability even with minor changes in the training dataset. The potentially unstable behaviors of a ML model decrease the model’s generalizability to unseen data, likely limiting its applicability as an assistant tool for decision making. Heterogeneous ensemble models are recognized to achieve greater generalizability compared to single models owing to their structural diversity. In this study, various machine learning (ML) models are employed to understand the relationship between environmental factors and benthic macroinvertebrate health. To obtain a model with better generalizability, the present study compares the generalizability of heterogeneous ensembles with those of homogeneous ensembles and single models by using the bias–variance decomposition. The models classified five grades (very good to very poor) of benthic macroinvertebrate index (BMI). The models incorporated diverse environmental factors, including water quality, hydrology, meteorological conditions, land cover, and stream properties, as input variables. The data were monitored at 2,915 sites in the four major river watersheds in South Korea during the 2016–2021 period. The results indicated better generalizability of the heterogeneous and homogeneous ensembles than single models. Moreover, heterogeneous ensembles tended to show higher generalizability than homogeneous ensembles, although the differences were marginal. Weighted soft voting was the most generalizable of the heterogeneous ensembles, with loss of 0.49. Weighted soft voting also delivered acceptable classification performance on the test set, with accuracy of 0.52. The identified contributions of the environmental factors to BMI predictions and the directions of their effects agreed with established knowledge, confirming the reliability of the predictions. These results demonstrate the usefulness of the heterogeneous ensemble models for increasing the generalizability of ML model predictions. Furthermore, despite the slightly lower generalizability than voting-based ensembles, homogeneous ensembles demonstrated comparable levels of generalizability to heterogeneous ensembles.

Reviewing ensemble classification methods in breast cancer

Predicting stock returns by classifier ensembles

EEG signal classification is a challenging task in that the nature of the EEG data may vary from subject to subject, and change over time for the same subject. To improve classification performance, we propose to construct heterogeneous classifier ensembles, where not only the base classifiers are of different types, but they have different input features as well. The classification performance of the proposed method has been examined on Berlin BCI competition III datasets IVa. Our comparative results clearly show that heterogeneous ensembles outperform single models as well as ensembles having the same input features.

Dynamic ensemble selection (or dynamic pruning) is a learning paradigm that reduces the size of an ensemble of classifiers for each test pattern as input. The purpose of this approach is to improve the performances of the ensemble method by eliminating dynamically all the weak learners for a specific input. As of now, most of the dynamic pruning methods use information about the validation data and intermediate performances of the ensemble's classifiers on it, but none of them take the advantage of the intrinsic information given by those classifiers. In this paper we propose a new dynamic ensemble selection framework based on a tree internal supervised measure. This implies that our approach works on homogeneous ensemble methods with trees as base learners. This approach is motivated by the fact that in high dimensional datasets, usual metrics like euclidean distance suffer from the curse of dimensionality. Moreover, a similarity measure based on decision tree structure can be considered as supervised, which is well suited for supervised learning. Our framework is called ST-DES for Similarity Tree Dynamic Ensemble Selection. In this work, we use one of the best ensemble algorithm, Random Forest, as the base ensemble generator. We then compare the performance in terms of accuracy of ST-DES against the best known approaches in literature on different UCI datasets. We show that our approach outperforms the existing ones especially for high dimensional datasets. Besides, we show that this approach is not disturbed by random noise in features.

The COVID-19 pandemic has necessitated the development of accurate and efficient classification models for diagnosis and prognosis. While deep learning has shown promising results in various medical applications, its combination with ensemble methods, which amalgamate the predictions of multiple classifiers, can further enhance the model's accuracy. This research paper introduces a novel approach called Deep Neural Ensemble Classification (DNEC) to tackle the challenge of Developing an enhanced ensemble model using deep learning algorithms and comparing its performance with existing ensemble methods. The research problem stems from the literature gap, where existing studies primarily focus on single-model approaches, lacking in-depth exploration of ensemble methods for COVID-19 classification. Motivated by the potential improvement in classification accuracy through ensemble methods, this study aims to create a deep neural ensemble classification model to improve the classification accuracy tailored for COVID-19 data. A set of diverse classifiers, including k-nearest neighbour (IBK), decision tree (J48), naïve bayes (NB), support vector machine (SVM), and sequential minimal optimization (SMO), are utilized in the proposed ensemble method. The accuracy improvement of the ensemble classifiers is evaluated using various metrics such as precision, recall, F1 score, confusion matrix, and processing time. The proposed method demonstrates that IBK+SVM+NB emerges as the top-performing deep neural ensemble classifier with an accuracy score of 99.29% and a total run time of 27.61 seconds. The innovative ensemble techniques introduced in this research contribute to the existing body of knowledge by filling the identified literature gap and offering a novel and highly accurate approach for COVID-19 classification.

Predicting Students at Risk of Early Dropping Out from Course Using Ensemble Classification Methods

Electricity load demand is the fundamental building block for all utilities planning. The load demand data has nonlinear and non-stationary characteristics, which make it difficult to be predicted accurately by just computational intelligence or ensemble methods. Ensemble methods like Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) is a powerful tool to forecast power load demand time series. Heterogeneous ensemble, a combination of two base models, will be distinct or more powerful in forecasting power load demand. In this paper, Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) method is hybridized with three computational intelligence-based predictors: support vector regression (SVR), artificial neural network (ANN) and random forest (RF). The basis of this paper was to conduct a comparative study on the accuracy of the forecasting result from using heterogeneous ensemble method to individual computational intelligence or ensemble method for four different horizons. The performances of the heterogeneous method are compared and discussed. It shows that heterogeneous method has outperformed the individual computational intelligence and ensemble methods. Possible future works are also recommended for power load demand forecasting.

The subject of this article is exploration of methods for identifying the status of a computer system. The purpose of the article is development of a method for classifying a computer system anomalous state based on ensemble methods. Task: To investigate the usage of algorithms for building decision trees: REPTree, Random Tree, J48, HoeffdingTree, DecisionStump and bagging and boosting decision tree ensembles to identify a computer system anomalous state by analyzing operating system events. The methods used are artificial intelligence, machine learning and ensemble classification methods. The following results were obtained: the methods of identifying the computer systems anomalous state based on ensemble methods were investigated, namely, bagging, boosting, and classifiers: REPTree, Random Tree, J48, HoeffdingTree, DecisionStump to identify a computer system anomalous state. The different classifiers set and classifiers ensembles were developed. Training and cross-validation on each algorithm was performed. The developed classifiers performance has been evaluated. The research suggests an ensemble method of a computer system state classifying based on the J48 decision tree algorithm. Conclusions. The scientific novelty of the obtained results consists in creating an ensemble method for classifying the state of a computer system based on a decision tree, which makes it possible to increase the reliability and speed of classification.

Educational data mining has illustrated an increasing demand for extracting and maneuvering data from academic backdrop, to generate prolific information which is indispensible for decision making. Therefore in this paper, an attempt has been made to deploy various data mining techniques including base and meta learning classifiers across our pedagogical dataset to foretell the performance of students. Among several contemporary ensemble approaches, researchers have practiced widespread learning classifiers viz. boosting to predict the performance of students. As exploitation of ensemble methods is considered to be significant phenomenon in classification and prediction mechanisms, therefore analogous method (boosting) has been applied across our pedagogical dataset. The entire results have been evaluated with 10-fold cross validation, once pedagogical dataset has been subjected to base classifiers including j48, random tree, naive bayes and knn. In addition, techniques such as oversampling (SMOTE) and undersampling (Spread subsampling) have been employed to further draw a comparison among ensemble classifiers and base classifiers. These methods were exploited with the key objective to observe any improvement in prediction accuracy of students.

A weighted multiple classifier framework based on random projection