A Selective Detector Ensemble for Concept Drift Detection

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

<title>Abstract</title> The occurrence of concept drift is an important phenomenon in an operational scenario related to machine learning based system which affects the performance of pre-trained models. To maintain the integrity and confidence in predictions, an effective drift detection and adaptation mechanism is a vital component in these operational systems. Apart from different supervised, semi-supervised and unsupervised drift detection techniques, recently the research trend has been focused more on deep learning-based methods specifically based on autoencoders. A hallmark of autoencoders is their versatility. Different types and implementations exist, each specializing in handling specific tasks. In case of drift detection, the focus is to learn the data distribution using autoencoder and to measure the deviation of the newly arriving data in terms of reconstruction loss. While standard or vanilla autoencoders are the most used; other variations also exist and can be evaluated for better drift detection performance. Apart from the type of the autoencoder, another important consideration is the use of right set of parameters and hyperparameters for an autoencoder based drift detection mechanism. In this paper, we provide a framework to optimize the performance of autoencoder based drift detection methods. We provide a theoretical and an empirical evaluation of other applicable types of autoencoders including denoising, variational and standard, followed by a detailed mechanism for selecting the best hyperparameters using grid search and selection of drift detection method’s specific parameters using both grid search and Bayesian Optimization (BO). For experimentation, we have used AEDDM (Autoencoder-based Drift Detection Method) as the base drift detection method to produce Opt-AEDDM- the optimized version of AEDDM. Detailed experiments on 4 synthetic (Hyperplane, Gaussian, VD and RBF) and two real world datasets (NOAA and Forest Covertype) prove the applicability and effectiveness of the proposed framework in finding the best autoencoder with best set of hyperparameters and parameters with improved drift detection performance.

Neural network models are widely adopted in real-world applications for processing streaming data. However, these applications often face challenges in terms of accuracy degradation, caused by changes in the data distribution of the stream data compared to the training data. Two underlying reasons contribute to these changes. The first, known as the concept drift problem, occurs when there is a change in the correlation between the input data and the prediction output, making the models trained on the training data no longer suitable for the new data. The second reason, known as the novelty problem, arises when real-world data contains unexpected data categories that were not present in the training data, resulting in incorrect predictions. The research community has divided into different groups and each developed various methods to detect either concept drift or novelty distribution changes. However, these methods only address one aspect of the problem and are unable to distinguish between them. This leads to an inappropriate allocation of model maintenance resources, including the high cost of model retraining and the acquisition of true label data. In this study, we aim to address this gap by proposing a novel concept drift detection method that is capable of distinguishing between known labeled concept drift and novelty. Our method is also more efficient than existing drift detection methods, making it suitable for applications on neural networks.

Introduction: This study addresses the challenge of concept drift in multivariate time series (MTS) forecasting, where data distributions evolve, degrading model performance. The objective is to propose an adaptive hybrid model that combines Long Short-Term Memory (LSTM) networks with the ADaptive WINdowing (ADWIN) algorithm for effective drift detection and adaptive forecasting in real-time environments. Methods: The proposed ADWIN-LSTM framework integrates Bi-LSTM layers for temporal sequence modelling and ADWIN for monitoring prediction residuals to detect concept drift. A dynamic sliding window mechanism adjusts the training data scope based on drift type. Data preprocessing includes normalization, STL-based detrending, and PCA for dimensionality reduction. Hyperparameters are optimized using grid search. Results: Experiments on real-world and synthetic MTS datasets demonstrate that the proposed model outperforms baseline models (LSTM, GRU, CNN-LSTM, and ADWIN-RF) across RMSE, MAE, MAPE, and R² metrics. The model detects both abrupt and gradual drifts with minimal false positives and low detection delay (≤ 5 steps). Post-drift adaptation significantly improves forecasting accuracy. Discussion: The adaptive retraining strategy triggered by drift detection ensures computational efficiency and robustness in volatile environments. The dynamic integration of forecasting and drift detection enhances model adaptability to evolving data distributions. Conclusion: The ADWIN-LSTM framework effectively combines predictive learning and realtime drift adaptation, making it suitable for dynamic, high-stakes environments such as energy, traffic, and environmental systems. Future work includes online optimization and deployment on streaming platforms.

Machine learning algorithms have been widely used in IoT application systems for predicting labels of input data streams from IoT sensors. Since data streams are not always stationary in real-world scenarios, underlying data distribution changes may cause a deterioration of the prediction performance that is known as concept drift. Existing concept drift detection methods often assume that complete and true labels for detecting the prediction errors are always available immediately after the prediction. However, such an assumption is not realistic in real IoT application systems. This paper experimentally investigates the robustness of six representative concept drift detectors against unreliable data streams containing the data with error labels and the data with no labels. Robustness is evaluated in terms of the average detection delay and precision by increasing the ratios of error-labeled and missing-label data in a synthetic data stream. Our experimental results show that Cumulative Sum (CUSUM), Page-Hinkley (PH), and Drift Detection Method (DDM) achieve relatively stable performances when the ratio of error-labeled data is less than 40%. With respect to the drift detection efficiency, CUSUM can be regarded as the most robust detector among other detectors tested in our experiments.

A comparative study on concept drift detectors

Concept drift referring to the changes of data distributions has been one critical challenge typically associated with mining data streams. Current drift detection and adaptation methods focus on how to immediately detect the distribution changes once the concept drift occurs and swiftly update the model to be applicable to the newly arrived data instances. Most of those methods assume the data does not have noise or the noise is too weak to affect the modeling procedure. However, realworld data are normally contaminated, and denoise techniques are highly preferred as a necessary preprocess. This issue is more complex for a data stream with concept drift because the noise is very likely to be confused with drift. Motivated by that, this paper proposes a Noise-tolerant Fuzzy c-means based drift Adaptation method (NFA) which can adapt to the changing distributions and is suitable for noisy data streams. The concept drift problem is solved by using a fuzzy c-means based regression model to continuously include the most relevant data instances to the latest pattern in the training set. In addition, a denoise technique is designed in NFA to remove noise, and the ability of incremental updating enables it to be embedded in the incremental drift adaptation process, and therefore NFA can solve concept drift and noise problems at the same time. Experimental evaluation results also show good performance of our method on handling data streams with concept drift and noise.

Most drift detection mechanisms designed for classification problems works in a post-event manner: after receiving the data set completely (patterns and class labels of the train and test set), they apply a sequence of procedures to identify some change in the class-conditional distribution - a concept drift. However, detecting changes after its occurrence can be in some situations harmful for the process under supervision. This paper proposes a pre-event approach for abrupt drift detection, called by A2D2. Briefly, this method is composed of three steps: (i) label the patterns from the test set, using an unsupervised method; (ii) compute some statistics from the train and test set, conditioned on the given class labels; and (iii) compare the train and test statistics using a multivariate hypothesis test. Also, it has been proposed a procedure for creating datasets with abrupt drift. This procedure was used in the sensivity analysis of A2D2, in order to understand the influence degree of each parameter on its final performance.

A large-scale comparison of concept drift detectors

Dynamic changes are a part of everyday life. When there is a change in data, the classification models need to be adaptive to the changes. In this paper we propose adaptive supervised learning model for training set selection under concept drift data streams. This paper focuses on adaptive supervised learning techniques, where adaptivity to changes in data over time is achieved by selective training set methodology. These selective training set methods typically can be used plugging in various base classifiers. In this work we consider accuracy (generalization error) as the primary performance measure for concept drift learners. In this paper our research follows the three main drift types, starting from sudden drift, via gradual drift to reoccurring concepts. We give methodological contributions to concept drift phenomenon under real time application i.e. Electricity pricing contexts and expected change types. In this paper, a proposed methodology consist of four algorithms, first algorithm i.e. Optimal Window Resizing Algorithm under sudden drift to determine the optimal window length at a given time, identify to what extent a change point is different from the start of the training window and how this difference can be used to improve the accuracy of an adaptive learner. Second algorithm i.e. Gradual Drift algorithm which would unify two selection criteria: similarity in time and feature space to improve accuracy of an adaptive learner. Third algorithm i.e. Reoccurring Concept drift where previously seen patterns reoccur, but it is not certain when exactly and in what form they will repeat. Last algorithm i.e. Dynamic drift detection. In comparison to other methods, our proposed algorithms are faster and memory-less, a requirement for streaming applications. A proposed methodology is tested on Elec2 data, we get less error rate.

Aim/Purpose: This paper describes how to use a multilayer perceptron to improve concept drift recovery in streaming environments. Background: Classifying instances in a data stream environment with concept drift is a challenging topic. The base learner must be adapted online to the current data. Several data mining algorithms have been adapted/used to this type of environment. In this study, two techniques are used to speed up the adaptation of an artificial neural network to the current data, increasing its predictive accuracy while detecting the concept drift sooner. Methodology: Experiments were performed to analyze how some techniques behave in different scenarios and compare them with other classifiers built to deal with data streams and concept drifts. Contribution: This study suggests two techniques to improve the classification results: an embedded concept drift detection method to identify when a change has occurred and setting the learning rate to a higher level whenever a new concept is being learned to give more weight to recent instances, with its value decreased over time. Findings: Results indicate that gradually reducing the learning rate with an embedded concept drift detector has better statistical results than other single classifiers built to deal with data streams and concept drifts. Recommendations for Practitioners: Based on the empirical results, this study provides recommendations on how to improve the multilayer perceptron in data stream environments suffering from concept drifts. Recommendation for Researchers: Researchers should conduct investigations to increase the number of base classifiers used in data stream environments and in situations where concept drifts occur. Impact on Society: The objective of this study is to increase the use of multilayer perceptrons in data stream environments suffering from concept drifts, as nowadays, Hoeffding Trees and Naive Bayes are the base classifiers mostly used. Future Research: Additional research includes adapting the online learning rate by increasing/decreasing it based on the performance of the Multilayer Perceptron. This scheme would allow the removal of parameters that must be set by the user, like learning rate upper bound and number of instances to return to the stable value.

In times of abundant streams of data, the only constant we can rely on is change itself. Given an existing classification model, it might be impractical or impossible to retrain when concept drift impairs the performance. In this paper, we propose an ensemble learning approach that fixes errors of a given model via patches: smaller, local models that are applied when the base model is likely to fail. These patches are added to the ensemble when the base learner shows decreased performance for one or multiple classes, which we discover by applying explicit drift detection. Each patch covers the drift of a subset of the classes. Via the drift detection method, the misclassification behavior of the base learner can be matched to an existing patch. This reduces the total amount of patches to the number of seen concept drifts, which is an advantage in the long run. Patches can be added to the ensemble and refined incrementally. The ensemble decision is conducted by a sequential classifier, which allows a high number of ensemble members that can be added incrementally. Experimental results show that ensemble patching performs well w.r.t. accuracy and adaptation speed.

Incremental learning with concept drift has often been tackled by ensemble methods, where models built in the past can be retrained to attain new models for the current data. Two design questions need to be addressed in developing ensemble methods for incremental learning with concept drift, i.e., which historical (i.e., previously trained) models should be preserved and how to utilize them. A novel ensemble learning method, namely, Diversity and Transfer-based Ensemble Learning (DTEL), is proposed in this paper. Given newly arrived data, DTEL uses each preserved historical model as an initial model and further trains it with the new data via transfer learning. Furthermore, DTEL preserves a diverse set of historical models, rather than a set of historical models that are merely accurate in terms of classification accuracy. Empirical studies on 15 synthetic data streams and 5 real-world data streams (all with concept drifts) demonstrate that DTEL can handle concept drift more effectively than 4 other state-of-the-art methods.

The object of the study is the adaptive machine learning systems that are able to process large amounts of rapidly changing streaming data in real time. The problem of maintaining prediction accuracy and computational efficiency in the presence of concept drift is treated. Concept drift refers to the overweighting of static models when stationary models are tried, and the nature of the underlying distributions changes. The adaptive architecture includes revision divergence-oriented concept drift detection, incremental model updating via hyper-dimensional statistical clustering of segments. Results from experiments using simulated and real-world datasets demonstrate that the adaptive architecture maintains predictive accuracy above 0.83 across abrupt, gradual, recurrent, and continuous drift scenarios. Compared with non-adaptive models, adaptation latency is reduced by approximately 2.6×, while unnecessary retraining operations are decreased by up to 40%. These results are possible due to the fact that proposed framework is able to retrain solutions if, and only if, distributional changes are determined to be statistically significant and meaningful to the model. This leads to the avoidance of processors being given redundant computations and providing a steady-state model during non-drift conditions. A principal contribution is that feature engineering is accomplished in a drift-aware manner, thresholding is made adaptive to the distributions indicated, and update mechanisms are employed which efficiently utilize resources in a unified high-performance streaming pipeline. The architecture performs well under abrupt, gradual, recurrent, and continuous drift and effective for real-time applications which include smart-city analytics, cyber security monitoring, roadways system works, and IoT for industrial systems

6G networks are envisioned to be trustworthy systems that can self-maintain their reliability and service availability. The ability to automatically recognize and predict any faults or failures that occur while delivering services is a step towards realizing such systems. Real-time analytics based on Artificial Intelligence (AI)/Machine Learning (ML) techniques provides the required functionality. However, in the highly dynamic and complex environment of 6G services, the distribution of data feeding to the ML models is subject to change over time (i.e., concept drift), which makes persistent model accuracy challenging without any frequent model retraining. To address this issue, in this paper, we propose a system that predicts the faults in an edge cloud environment, where the prediction models are trained and automatically adapted to the concept drifts via Transfer Learning (TL). To present the effectiveness of our system, we implemented an edge cloud testbed and introduced CPU over-utilization and network congestion fault to it in the presence of concept drift. We used Convolutional Neural Network (CNN), Long-Short Term Memory (LSTM), and a combination of them as fault prediction models. For detecting the drifts, we have implemented and compared four drift detection methods, and we utilized three TL scenarios for adapting the model to the drift. Our results indicate the superiority of our system in maintaining a persistent accuracy in the presence of concept drift compared to a prediction system without a drift handling entity.