An Experimental Comparison of Ensemble Classifiers for Evolving Data Streams

In real-time streaming data, concept drift and class imbalance may occur simultaneously which causes the performance degradation of the online machine learning models. Most of the existing work is limited to addressing these issues for binary class data streams. Very little focus is given to the multi-class data streams. The most common approach to address these issues is ensemble learning. Ensemble learning consists of multiple classifiers combined which are trained on different subsets of the data to improve the overall accuracy. The performance of the ensemble learning approach suffers in case the new classifier is not trained on appropriate data (the data about the new concept). To address this gap, this study has proposed a Smart Adaptive Ensemble Model (SAEM) to address the issues of concept drift and class imbalance for multi-class data streams. The SAEM monitors the feature-level change in data distribution and creates a background ensemble to train the new classifier on features that observe change. To address the class imbalance issue, SAEM applies higher weights on the minority class instances using the dynamic class imbalance ratio. The proposed model outperformed the existing state-of-the-art approaches on the eight different data streams. The results showed an average improvement of 15.857% in accuracy, 20.35% in Kappa, 16.12% in F1-score, 15.58% in precision, and 16.42% in recall. The Friedman test confirmed statistically significant performance differences among all models across five key metrics. Based on the obtained results, the research findings strongly support the notion that SAEM exhibits enhanced effectiveness and efficiency as a solution for online learning applications.

New challenges have emerged in data mining as the traditional techniques have floundered with real-time data streams. The traditional technique needs refurbishing so as to acclimatize with concept drifting data streams. Thus dealing with the concept changes is the most imperative task of stream data mining. Ensemble classifiers have the ability to automatically adapt with the incoming drifts and, therefore, it is the most interesting research area in data stream mining. Bagging, Boosting and Random forest generation are the common ensemble techniques and are the most popular machine learning approaches in the current scenario for static data (Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9–10):469–1495, [1]). A large number of base classifiers in an ensemble can cause computational overhead. Data mining classifiers for real-time data streams, therefore, need to be updated constantly and retrained with the labeled instances of the newly arrived novel classes in data streams and to cope with concept drift; otherwise, the mining models will become less and less accurate as time passes by. However, for data streams, adaptive random forest algorithms have been widely used for ensemble generation due to its competence to handle different types of drifts. This paper proposes a modified adaptive random forest with meta level learner algorithm and concept adaptive very fast decision tree to overcome the concept drift problem in real-time data streams. The proposed algorithm is experimentally compared with state-of-the-art adaptive random forest algorithm on several real synthetic datasets. Results indicate its efficiency in terms of accuracy and processing time.

Class imbalance poses new challenges when it comes to classifying data streams. Many algorithms recently proposed in the literature tackle this problem using a variety of data-level, algorithm-level, and ensemble approaches. However, there is a lack of standardized and agreed-upon procedures and benchmarks on how to evaluate these algorithms. This work proposes a standardized, exhaustive, and comprehensive experimental framework to evaluate algorithms in a collection of diverse and challenging imbalanced data stream scenarios. The experimental study evaluates 24 state-of-the-art data streams algorithms on 515 imbalanced data streams that combine static and dynamic class imbalance ratios, instance-level difficulties, concept drift, real-world and semi-synthetic datasets in binary and multi-class scenarios. This leads to a large-scale experimental study comparing state-of-the-art classifiers in the data stream mining domain. We discuss the advantages and disadvantages of state-of-the-art classifiers in each of these scenarios and we provide general recommendations to end-users for selecting the best algorithms for imbalanced data streams. Additionally, we formulate open challenges and future directions for this domain. Our experimental framework is fully reproducible and easy to extend with new methods. This way, we propose a standardized approach to conducting experiments in imbalanced data streams that can be used by other researchers to create complete, trustworthy, and fair evaluation of newly proposed methods. Our experimental framework can be downloaded from https://github.com/canoalberto/imbalanced-streams.

To address the critical industry pain points in streaming data scenarios across industrial production, intelligent transportation and other related fields—namely continuous performance degradation of classification models induced by concept drift, the absence of dedicated closed-loop drift adaptation processing capability in mainstream stream processing systems, poor engineering implementability of specialized tools, and the inherent difficulty in balancing real-time performance and classification accuracy—this paper designs and implements a Dynamic Adaptive Concept Drift Processing System (DACPS) for streaming data classification. The system adopts a hierarchical modular architecture, integrating five core functional modules: streaming data preprocessing, real-time concept drift detection, dynamic sample optimization, incremental model training, and full-process management and control. This architecture enables fully automated and adaptive processing for the end-to-end workflow of streaming data classification and prediction. Extensive multi-dimensional functional tests, specialized performance tests, and scenario-based validation demonstrate that the system can stably adapt to diverse types of concept drift scenarios, and meets all design specifications in terms of classification accuracy, convergence speed, operating efficiency, and resource occupation control. Meanwhile, the system is equipped with an accessible visual interactive interface, making it widely adaptable to the streaming data classification and processing requirements across multiple application domains.

Nowadays, many physical world appliances found data streams like telecommunication system, multimedia data, medical data streams. Traditional data stream mining allows storage of data and multiple scan of dataset. But it is next to impossible to save or scan it more than one or two times, because of its mountainous size. It is essential to develop the processing systems which scans once and examines the methods. Because of this, data stream mining becomes an emerging topic for research in knowledge discovery. Effective classification of such data streams finds many stream mining provocations like immeasurable length, increment learning, concept drift. So, we have to either update existing mining classifiers or generate a new technique for data stream classification. In this paper, we point out three different classification methods of decision tree called Hoeffding tree, VFDT, and CVFDT, which focuses on these classification problems.

Radio wave data gathered by pulsar finding telescopes are required to be classified while being streamed. The reason for that is the practical constraints of traditional machine learning algorithms on streaming datasets. Traditional machine learning algorithms would take considerable compute power, memory and time to give pragmatic results.(recent surveys collect data at the rate of 0.5 – 1 terabyte per second) Stream classification algorithms are specifically developed to address the above limitations and can classify data streams without taking up a lot of memory or training time. They relate with characteristics of data streams such as concept drift and limited memory. Extremely Fast Decision Tree is one of the stream classification algorithms that can learn incrementally when it sees new data. However, data from pulsar detecting datastreams are highly imbalanced (there are less examples of pulsars in the data than non-pulsar objects). Learning incrementally from such a datastream would be a destructive interference for the model’s precision (of detecting pulsars). In this research, we introduce an improved version of the Extremely Fast Decision Tree, that is able to learn imbalanced data streams. Our approach is fast, accurate, and avoids the pitfalls of class imbalance and concept drift.

Ensemble classification technique is great advantage over conventional classifier such as statistical, binary and neural network classifier. Ensemble technique improves the performance of other classifier with some valid constraints. In the improvement of ensemble classifier cluster play important role for data grouping before classification. Cluster oriented ensemble classifier maintain and control the diversity of data during classification process. The diversity of cluster oriented ensemble classifier implied in stream data classification. Stream data classification is critical task due to diversity of data. The critical task of diversity such as infinite population, data drift and feature evaluation technique is overcome through ensemble classifier. Various author proposed a method for stream data classification using ensemble technique. In this paper we give the review of ensemble technique used in stream data classification with clustering technique.

Today's ever‐increasing generation of streaming data demands novel data mining approaches tailored to mining dynamic data streams. Data streams are non‐static in nature, continuously generated, and endless. They often suffer from class imbalance and undergo temporal drift. To address the classification of consecutive data instances within imbalanced data streams, this research introduces a new ensemble classification algorithm called Rarity Updated Ensemble with Oversampling (RUEO). The RUEO approach is specifically designed to exhibit robustness against class imbalance by incorporating an imbalance‐specific criterion to assess the efficacy of the base classifiers and employing an oversampling technique to reduce the imbalance in the training data. The RUEO algorithm was evaluated on a set of 20 data streams and compared against 14 baseline algorithms. On average, the proposed RUEO algorithm achieves an average‐accuracy of 0.69 on the real‐world data streams, while the chunk‐based algorithms AWE, AUE, and KUE achieve average‐accuracies of 0.48, 0.65, and 0.66, respectively. The statistical analysis, conducted using the Wilcoxon test, reveals a statistically significant improvement in average‐accuracy for the proposed RUEO algorithm when compared to 12 out of the 14 baseline algorithms. The source code and experimental results of this research work will be publicly available at https://github.com/vkiani/RUEO.

<abstract><p>With the widespread application of smart devices, the security of internet of things (IoT) systems faces entirely new challenges. The IoT data stream operates in a non-stationary, dynamic environment, making it prone to concept drift. This paper focused on addressing the issue of concept drift in data streams, with a key emphasis on introducing an innovative drift detection method-ensemble multiple non-parametric concept localization detectors, abbreviated as EMNCD. EMNCD employs an ensemble of non-parametric statistical methods, including the Kolmogorov-Smirnov, Wilcoxon rank sum and Mann-Kendall tests. By comparing sample distributions within a sliding window, EMNCD accurately detects concept drift, achieving precise localization of drift points, and enhancing overall detection reliability. Experimental results demonstrated the superior performance of EMNCD compared to classical methods on artificial datasets. Simultaneously, to enhance the robustness of data stream processing, we presented an online anomaly detection method based on the isolation forest (iForest). Additionally, we proposedwhale optimization algorithm (WOA)-extreme gradient boosting (XGBoost), a drift adaptation model employing XGBoost as a base classifier. This model dynamically updates using drift points detected by EMNCD and fine-tunes parameters through the WOA. Real-world applications on the edge-industrial IoTset (IIoTset) intrusion dataset explore the impact of concept drift on intrusion detection, where IIoT is a subclass of IoT. In summary, this paper focused on EMNCD, introducing innovative approaches for drift detection, anomaly detection, and drift adaptation. The research provided practical and viable solutions to address concept drift in data streams, enhancing security in IoT systems.</p></abstract>

Imbalanced streaming data is commonly encountered in real-world data mining and machine learning applications, and has attracted much attention in recent years. Both imbalanced data and streaming data in practice are normally encountered together; however, little research work has been studied on the two types of data together. In this paper, we propose a multi-window based ensemble learning method for the classification of imbalanced streaming data. Three types of windows are defined to store the current batch of instances, the latest minority instances, and the ensemble classifier. The ensemble classifier consists of a set of latest sub-classifiers, and the instances employed to train each sub-classifier. All sub-classifiers are weighted prior to predicting the class labels of newly arriving instances, and new sub-classifiers are trained only when the precision is below a predefined threshold. Extensive experiments on synthetic datasets and real-world datasets demonstrate that the new approach can efficiently and effectively classify imbalanced streaming data, and generally outperforms existing approaches.

Genetic Programming is an evolutionary soft computing approach. Data streams are the order of the day input sources. In general, data streams exhibit a peculiar behavior of drifting the concepts as time passes by. Here is a study of GP Classifier on Concept Drifting Data Streams. GP classifier performance is compared to that of other state-of-the-art data mining and stream classification approaches. Boosting is a machine learning meta-algorithm for performing supervised learning. A weak learner is defined to be a classifier which is only slightly correlated with the true classification. In contrast, a strong learner is a classifier that is arbitrarily well-correlated with the true classification. Boosting combines a set of weak learners to create a strong learner. It is observed that the Boosting GP approach is beating Boosting Naive Bayes classification on Concept Drifting Data Streams. Hence it is found that GP is a competent algorithm for Concept Drifting Data Stream classification.

Concept drift (CD) in data streaming scenarios such as networking intrusion detection systems (IDS) refers to the change in the statistical distribution of the data over time. There are five principal variants related to CD: incremental, gradual, recurrent, sudden, and blip. Genetic programming combiner (GPC) classification is an effective core candidate for data stream classification for IDS. However, its basic structure relies on the usage of traditional static machine learning models that receive onetime training, limiting its ability to handle CD. To address this issue, we propose an extended variant of the GPC using three main components. First, we replace existing classifiers with alternatives: online sequential extreme learning machine (OSELM), feature adaptive OSELM (FA-OSELM), and knowledge preservation OSELM (KP-OSELM). Second, we add two new components to the GPC, specifically, a data balancing and a classifier update. Third, the coordination between the sub-models produces three novel variants of the GPC: GPC-KOS for KA-OSELM; GPC-FOS for FA-OSELM; and GPC-OS for OSELM. This article presents the first data stream-based classification framework that provides novel strategies for handling CD variants. The experimental results demonstrate that both GPC-KOS and GPC-FOS outperform the traditional GPC and other state-of-the-art methods, and the transfer learning and memory features contribute to the effective handling of most types of CD. Moreover, the application of our incremental variants on real-world datasets (KDD Cup '99, CICIDS-2017, CSE-CIC-IDS-2018, and ISCX '12) demonstrate improved performance (GPC-FOS in connection with CSE-CIC-IDS-2018 and CICIDS-2017; GPC-KOS in connection with ISCX2012 and KDD Cup '99), with maximum accuracy rates of 100% and 98% by GPC-KOS and GPC-FOS, respectively. Additionally, our GPC variants do not show superior performance in handling blip drift.

“Concept drift” and class imbalance are two challenges for supervised classifiers. “Concept drift” (or non-stationarity) is changes in the underlying function being learnt, and class imbalance is a vast difference between the numbers of instances in different classes of data. Class imbalance is an obstacle for the efficiency of most classifiers. Previous methods for classifying non-stationary and imbalanced data streams mainly focus on batch solutions, in which the classification model is trained using a chunk of data. Here, we propose an online Neural Network (NN) model. The NN model, is composed of two different parts for handling concept drift and class imbalance. Concept drift is handled with a forgetting function and class imbalance is handled with a specific error function which assigns different importance to error in separate classes. The proposed method is evaluated on 3 synthetic and 8 real world datasets. The results show statistically significant improvement to previous online NN methods.

Online class imbalance learning constitutes a new problem and an emerging research topic that focusses on the challenges of online learning under class imbalance and concept drift. Class imbalance deals with data streams that have very skewed distributions while concept drift deals with changes in the class imbalance status. Little work exists that addresses these challenges and in this paper we introduce queue-based resampling, a novel algorithm that successfully addresses the co-existence of class imbalance and concept drift. The central idea of the proposed resampling algorithm is to selectively include in the training set a subset of the examples that appeared in the past. Results on two popular benchmark datasets demonstrate the effectiveness of queue-based resampling over state-of-the-art methods in terms of learning speed and quality.

Online semi-supervised active learning ensemble classification for evolving imbalanced data streams