Learning Decision Trees from Data Streams with Concept Drift

New challenges have emerged in data mining as the traditional techniques have floundered with real-time data streams. The traditional technique needs refurbishing so as to acclimatize with concept drifting data streams. Thus dealing with the concept changes is the most imperative task of stream data mining. Ensemble classifiers have the ability to automatically adapt with the incoming drifts and, therefore, it is the most interesting research area in data stream mining. Bagging, Boosting and Random forest generation are the common ensemble techniques and are the most popular machine learning approaches in the current scenario for static data (Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9–10):469–1495, [1]). A large number of base classifiers in an ensemble can cause computational overhead. Data mining classifiers for real-time data streams, therefore, need to be updated constantly and retrained with the labeled instances of the newly arrived novel classes in data streams and to cope with concept drift; otherwise, the mining models will become less and less accurate as time passes by. However, for data streams, adaptive random forest algorithms have been widely used for ensemble generation due to its competence to handle different types of drifts. This paper proposes a modified adaptive random forest with meta level learner algorithm and concept adaptive very fast decision tree to overcome the concept drift problem in real-time data streams. The proposed algorithm is experimentally compared with state-of-the-art adaptive random forest algorithm on several real synthetic datasets. Results indicate its efficiency in terms of accuracy and processing time.

Long short-term memory self-adapting online random forests for evolving data stream regression

Decision tree has become one of the most accepted tools for mining data streams after Hoeffding tree was anticipated in the literature. The most vital point of constructing the decision tree is to find out the best attribute to split the considered node. Numerous methods to resolve this problem were presented so far, however, there are some shortcomings such that they are either mathematically not justified or time-consuming. In this paper, a new decision tree algorithm named Adaptive Gaussian Decision Tree (AGDT) is proposed that employs a statistical method (Gaussian Bound) for determining the best attribute in a node. This statistical method ensures that the finest attribute chosen in the considered node using a limited data sample is the same as it would be in the case of the complete data stream hence solving the storage constraints associated with the data stream. In order to handle the concept drift problems AGDT uses fixed-size window to determine which nodes are aging and may need updating. During implementation it is observed that the proposed decision tree method provides a greater classification accuracy, compared with the existing algorithm with the same probability for concept drift data streams.

Aim/Purpose: This paper describes how to use a multilayer perceptron to improve concept drift recovery in streaming environments. Background: Classifying instances in a data stream environment with concept drift is a challenging topic. The base learner must be adapted online to the current data. Several data mining algorithms have been adapted/used to this type of environment. In this study, two techniques are used to speed up the adaptation of an artificial neural network to the current data, increasing its predictive accuracy while detecting the concept drift sooner. Methodology: Experiments were performed to analyze how some techniques behave in different scenarios and compare them with other classifiers built to deal with data streams and concept drifts. Contribution: This study suggests two techniques to improve the classification results: an embedded concept drift detection method to identify when a change has occurred and setting the learning rate to a higher level whenever a new concept is being learned to give more weight to recent instances, with its value decreased over time. Findings: Results indicate that gradually reducing the learning rate with an embedded concept drift detector has better statistical results than other single classifiers built to deal with data streams and concept drifts. Recommendations for Practitioners: Based on the empirical results, this study provides recommendations on how to improve the multilayer perceptron in data stream environments suffering from concept drifts. Recommendation for Researchers: Researchers should conduct investigations to increase the number of base classifiers used in data stream environments and in situations where concept drifts occur. Impact on Society: The objective of this study is to increase the use of multilayer perceptrons in data stream environments suffering from concept drifts, as nowadays, Hoeffding Trees and Naive Bayes are the base classifiers mostly used. Future Research: Additional research includes adapting the online learning rate by increasing/decreasing it based on the performance of the Multilayer Perceptron. This scheme would allow the removal of parameters that must be set by the user, like learning rate upper bound and number of instances to return to the stable value.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

The advances in computing software, hardware, connected devices and wireless communication infrastructure in recent years have led to the desire to work with streaming data sources. Yet the number of techniques, approaches and algorithms which can work with data from a streaming source is still very limited, compared with batched data. Although data mining techniques have been a well-studied topic of knowledge discovery for decades, many unique properties as well as challenges in learning from a data stream have not been considered properly due to the actual presence of and the real needs to mine information from streaming data sources. This thesis aims to contribute to the knowledge by developing a rule-based algorithm to specifically learn classification rules from data streams, with the learned rules are expressive so that a human user can easily interpret the concept and rationale behind the predictions of the created model. There are two main structures to represent a classification model; the ‘tree-based’ structure and the ‘rule-based’ structure. Even though both forms of representation are popular and well-known in traditional data mining, they are different when it comes to interpretability and quality of models in certain circumstances. The first part of this thesis analyses background work and relevant topics in learning classification rules from data streams. This study provides information about the essential requirements to produce high quality classification rules from data streams and how many systems, algorithms and techniques related to learn the classification of a static dataset are not applicable in a streaming environment. The second part of the thesis investigates at a new technique to improve the efficiency and accuracy in learning heuristics from numeric features from a streaming data source. The computational cost is one of the important factors to be considered for an effective and practical learning algorithm/system because of the needs to learn from continuous arrivals of data examples sequentially and discard the seen data examples. If the computing cost is too expensive, then one may not be able to keep pace with the arrival of high velocity and possibly unbound data streams. The proposed technique was first discussed in the context of the use of Gaussian distribution as heuristics for building rule terms on numeric features. Secondly, empirical evaluation shows the successful integration of the proposed technique into an existing rule-based algorithm for the data stream, eRules. Continuing on the topic of a rule-based algorithm for classification data streams, the use of Hoeffding’s Inequality addresses another problem in learning from a data stream, namely how much data should be seen from a data stream before starting learning and how to keep the model updated over time. By incorporating the theory from Hoeffding’s Inequality, this study presents the Hoeffding Rules algorithm, which can induce modular rules directly from a streaming data source with dynamic window sizes throughout the learning period to ensure the efficiency and robustness towards the concept drifts. Concept drift is another unique challenge in mining data streams which the underlying concept of the data can change either gradually or abruptly over time and the learner should adapt to these changes as quickly as possible. This research focuses on the development of a rule-based algorithm, Hoeffding Rules, for data stream which considers streaming environments as primary data sources and addresses several unique challenges in learning rules from data streams such as concept drifts and computational efficiency. This knowledge facilitates the need and the importance of an interpretable machine learning model; applying new studies to improve the ability to mine useful insights from potentially high velocity, high volume and unbounded data streams. More broadly, this research complements the study in learning classification rules from data streams to address some of the unique challenges in data streams compared with conventional batch data, with the knowledge necessary to systematically and effectively learn expressive and modular classification rules from data streams.

In data stream mining, one of the biggest challenges is to develop algorithms that deal with the changing data. As data evolve over time, static models become outdated. This phenomenon is called concept drift, and it is investigated extensively in the literature. Detecting and subsequently adapting to concept drifts yield more robust and better performing models. In this study, we present an unsupervised method called D3 which uses a discriminative classifier with a sliding window to detect concept drift by monitoring changes in the feature space. It is a simple method that can be used along with any existing classifier that does not intrinsically have a drift adaptation mechanism. We experiment on the most prevalent concept drift detectors using 8 datasets. The results demonstrate that D3 outperforms the baselines, yielding models with higher performances on both real-world and synthetic datasets.

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

Concept drift referring to the changes of data distributions has been one critical challenge typically associated with mining data streams. Current drift detection and adaptation methods focus on how to immediately detect the distribution changes once the concept drift occurs and swiftly update the model to be applicable to the newly arrived data instances. Most of those methods assume the data does not have noise or the noise is too weak to affect the modeling procedure. However, realworld data are normally contaminated, and denoise techniques are highly preferred as a necessary preprocess. This issue is more complex for a data stream with concept drift because the noise is very likely to be confused with drift. Motivated by that, this paper proposes a Noise-tolerant Fuzzy c-means based drift Adaptation method (NFA) which can adapt to the changing distributions and is suitable for noisy data streams. The concept drift problem is solved by using a fuzzy c-means based regression model to continuously include the most relevant data instances to the latest pattern in the training set. In addition, a denoise technique is designed in NFA to remove noise, and the ability of incremental updating enables it to be embedded in the incremental drift adaptation process, and therefore NFA can solve concept drift and noise problems at the same time. Experimental evaluation results also show good performance of our method on handling data streams with concept drift and noise.

Recent research shows that rule based models perform well while classifying large data sets such as data streams with concept drifts. A genetic algorithm is a strong rule based classification algorithm which is used only for mining static small data sets. If the genetic algorithm can be made scalable and adaptable by reducing its I/O intensity, it will become an efficient and effective tool for mining large data sets like data streams. In this paper a scalable and adaptable online genetic algorithm is proposed to mine classification rules for the data streams with concept drifts. Since the data streams are generated continuously in a rapid rate, the proposed method does not use a fixed static data set for fitness calculation. Instead, it extracts a small snapshot of the training example from the current part of data stream whenever data is required for the fitness calculation. The proposed method also builds rules for all the classes separately in a parallel independent iterative manner. This makes the proposed method scalable to the data streams and also adaptable to the concept drifts that occur in the data stream in a fast and more natural way without storing the whole stream or a part of the stream in a compressed form as done by the other rule based algorithms. The results of the proposed method are comparable with the other standard methods which are used for mining the data streams.

Nowadays, many physical world appliances found data streams like telecommunication system, multimedia data, medical data streams. Traditional data stream mining allows storage of data and multiple scan of dataset. But it is next to impossible to save or scan it more than one or two times, because of its mountainous size. It is essential to develop the processing systems which scans once and examines the methods. Because of this, data stream mining becomes an emerging topic for research in knowledge discovery. Effective classification of such data streams finds many stream mining provocations like immeasurable length, increment learning, concept drift. So, we have to either update existing mining classifiers or generate a new technique for data stream classification. In this paper, we point out three different classification methods of decision tree called Hoeffding tree, VFDT, and CVFDT, which focuses on these classification problems.

Data stream mining has been receiving increased attention due to its presence in a wide range of applications, such as sensor networks, banking, and telecommunication. One of the most important challenges during learning from data streams is reacting to concept drift, i.e. unforeseen changes of the stream's underlying data distribution. Traditional methods always used online learning to handle the concept drift problem. However, online learning requires high time cost during online training. To overcome this shortcoming, this paper proposes a Kalman filtering approach, which can provide robust concept drift detection, to track concept drift. Once concept drift happens, the online extreme learning machine is applied to update the tracking model, whereas the offline extreme learning machine is used when no concept drift occurs. Based on this idea, we propose a fusion framework to combine online and offline extreme learning machine to efficiently track the data stream. The experiment results indicate the superior performance of our method.

A frequent problem in data stream mining is concept drift, meaning the data distribution changes over time. A common issue when dealing with concept drift is insufficient data. Real-world applications of data stream mining often involve multiple data streams. However, most concept drift methods handle these data streams separately. This study uses data from other data streams to handle the problem of insufficient data. We propose a novel Multistream Concept Drift Handling Framework via data sharing, containing a fuzzy membership-based drift detection (FMDD) component and a fuzzy membership-based drift adaptation (FMDA) component, to train the new learning model for drifting streams by sharing weighted data from other nondrifting streams. A stream fuzzy set is defined with membership functions that measure the degree to which samples belong to a data stream. Our Concept Drift Handling Framework can detect when and in which stream concept drift occurs, and therefore the insufficient data issue can be solved by adding the weighted data from nondrifting streams to train new learning models. Synthetic and real-world experimental results show that our method can help avoid the insufficient data issue and thereby significantly improve the prediction performance.

The major challenge in mining data streams is the issue of concept drift, the tendency of the underlying data generation process to change over time. In this paper, we propose a general rule learning framework that can efficiently handle concept-drifting data streams and maintain a highly accurate classification model. The main idea is to focus on partial drifts by allowing individual rules to monitor the stream and detect if there is a drift in the regions they cover. A rule quality measure then decides whether the affected rules are inconsistent with the concept drift. The model is accordingly updated to only include rules that are consistent with the newly arrived concept. A dynamically maintained set of instances deemed relevant to the most recent concept is also kept at memory. Learning a new concept from a larger set of instances reduces the variance of data distribution and allows for a more accurate, stable classification model. Our experiments show that this approach not only handles the drift efficiently, but it also can provide higher classification accuracy compared to other competitive approaches on a variety of real and synthetic data sets.

<abstract><p>With the widespread application of smart devices, the security of internet of things (IoT) systems faces entirely new challenges. The IoT data stream operates in a non-stationary, dynamic environment, making it prone to concept drift. This paper focused on addressing the issue of concept drift in data streams, with a key emphasis on introducing an innovative drift detection method-ensemble multiple non-parametric concept localization detectors, abbreviated as EMNCD. EMNCD employs an ensemble of non-parametric statistical methods, including the Kolmogorov-Smirnov, Wilcoxon rank sum and Mann-Kendall tests. By comparing sample distributions within a sliding window, EMNCD accurately detects concept drift, achieving precise localization of drift points, and enhancing overall detection reliability. Experimental results demonstrated the superior performance of EMNCD compared to classical methods on artificial datasets. Simultaneously, to enhance the robustness of data stream processing, we presented an online anomaly detection method based on the isolation forest (iForest). Additionally, we proposedwhale optimization algorithm (WOA)-extreme gradient boosting (XGBoost), a drift adaptation model employing XGBoost as a base classifier. This model dynamically updates using drift points detected by EMNCD and fine-tunes parameters through the WOA. Real-world applications on the edge-industrial IoTset (IIoTset) intrusion dataset explore the impact of concept drift on intrusion detection, where IIoT is a subclass of IoT. In summary, this paper focused on EMNCD, introducing innovative approaches for drift detection, anomaly detection, and drift adaptation. The research provided practical and viable solutions to address concept drift in data streams, enhancing security in IoT systems.</p></abstract>