Revisiting streaming anomaly detection: benchmark and evaluation

Data stream mining has been receiving increased attention due to its presence in a wide range of applications, such as sensor networks, banking, and telecommunication. One of the most important challenges in learning from data streams is reacting to concept drift, i.e., unforeseen changes of the stream's underlying data distribution. Several classification algorithms that cope with concept drift have been put forward, however, most of them specialize in one type of change. In this paper, we propose a new data stream classifier, called the Accuracy Updated Ensemble (AUE2), which aims at reacting equally well to different types of drift. AUE2 combines accuracy-based weighting mechanisms known from block-based ensembles with the incremental nature of Hoeffding Trees. The proposed algorithm is experimentally compared with 11 state-of-the-art stream methods, including single classifiers, block-based and online ensembles, and hybrid approaches in different drift scenarios. Out of all the compared algorithms, AUE2 provided best average classification accuracy while proving to be less memory consuming than other ensemble approaches. Experimental results show that AUE2 can be considered suitable for scenarios, involving many types of drift as well as static environments.

Long short-term memory self-adapting online random forests for evolving data stream regression

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

Selective ensemble-based online adaptive deep neural networks for streaming data with concept drift

The data collected by sensors is streaming data in the Internet of Things(IoT). Although existing deep learningbased anomaly detection methods generally perform well on static data, they struggle to respond timely to streaming data after distribution changes. However, streaming data suffers from conceptual drift due to the highly dynamic nature of IoT. In network security, concept drift-oriented anomaly detection is a crucial task, because it can adjust the model to adapt to the latest data, and detect attacks in time. Existing streaming anomaly detection methods are confronted with some challenges, including of the latency of model updates, the uneven importance of new data, and the self-poisoning due to model self-updates. To tackle the above challenges, we propose a knowledge distillation based adaptive anomaly detection model towards concept-drift, ADTCD. ADTCD transfers the knowledge of the teacher model to the student model and only updates the student model to reduce the delay. We construct an algorithm of dynamically adjusting model parameters, which dynamically adjusts model weights through local inference on new samples, in order to improving the models responsiveness to new distribution data, meanwhile solving the problem of uneven importance of new data. In additions, we adopt a one-class support vector-based outlier removal method to tackle the self-poisoning problem. In comprehensive experiments on seven high-dimensional datasets, ADTCD achieves an AUC improvement of 12.46% compared to the state-of-the-art streaming anomaly detection methods. Our future direction will focus on exploring the concept drift problem using methods beyond autoencoders.

In machine learning, concept drift can cause the optimal solution to a given problem to change as time passes, leading to less accurate predictions. Concept drift can be sudden, gradual or reoccuring. Understanding the consequences of concept drift is particularly important in human-centric applications where changes in the underlying data and environment are common and unexpected. In order to gain a better understanding of the adverse effects of different types of concept drift on learners, we propose a novel simulation tool that is able to incrementally generate datasets with customisable concept drift by interacting with a human in a game-like setting. We illustrate our approach by generating and analysing concept drift simulations inspired by body-sensor based long-term activity recognition. Our initial results show that current unsupervised adaptation techniques can be caught in cyclic mislabelling and that a hybrid solution that is self-calibrating and semi-supervised is more robust than any of the two taken separately for this example.

A frequent problem in data stream mining is concept drift, meaning the data distribution changes over time. A common issue when dealing with concept drift is insufficient data. Real-world applications of data stream mining often involve multiple data streams. However, most concept drift methods handle these data streams separately. This study uses data from other data streams to handle the problem of insufficient data. We propose a novel Multistream Concept Drift Handling Framework via data sharing, containing a fuzzy membership-based drift detection (FMDD) component and a fuzzy membership-based drift adaptation (FMDA) component, to train the new learning model for drifting streams by sharing weighted data from other nondrifting streams. A stream fuzzy set is defined with membership functions that measure the degree to which samples belong to a data stream. Our Concept Drift Handling Framework can detect when and in which stream concept drift occurs, and therefore the insufficient data issue can be solved by adding the weighted data from nondrifting streams to train new learning models. Synthetic and real-world experimental results show that our method can help avoid the insufficient data issue and thereby significantly improve the prediction performance.

We present a novel method for concept drift detection, based on: 1) the development and continuous updating of online sequential extreme learning machines (OS-ELMs) and 2) the quantification of how much the updated models are modified by the newly collected data. The proposed method is verified on two synthetic case studies regarding different types of concept drift and is applied to two public real-world data sets and a real problem of predicting energy production from a wind plant. The results show the superiority of the proposed method with respect to alternative state-of-the-art concept drift detection methods. Furthermore, updating the prediction model when the concept drift has been detected is shown to allow improving the overall accuracy of the energy prediction model and, at the same time, minimizing the number of model updatings.

In this article, we propose an online and unsupervised anomaly detection algorithm for streaming data using an array of sliding windows and the probability density-based descriptors (PDDs) (based on these windows). This algorithm mainly consists of three steps: 1) we use a main sliding window over streaming data and segment this window into an array of nonoverlapping subwindows; 2) we propose the PDDs with dimension reduction, based on the kernel density estimation, to estimate the probability density of data in each subwindow; and 3) we design the distance-based anomaly detection rule to determine whether the current observation is anomalous. The experimental results and performances are presented based on the Numenta anomaly benchmark. Compared with the anomaly detection algorithm using the hierarchical temporal memory proposed by Numenta (which outperforms a wide range of other anomaly detection algorithms), our algorithm can perform better in many cases, that is, with higher detection rates and earlier detection for contextual anomalies and concept drifts.

Many real-world applications adopt multi-label data streams as the need for algorithms to deal with rapidly changing data increases. Changes in data distribution, also known as concept drift, cause existing classification models to rapidly lose their effectiveness. To assist the classifiers, we propose a novel algorithm called Label Dependency Drift Detector (LD3), an unsupervised concept drift detector using label dependencies within the data for multi-label data streams. Our study exploits the dynamic temporal dependencies between labels using a label influence ranking method, which leverages a data fusion algorithm and uses the produced ranking to detect concept drift. LD3 is the first unsupervised concept drift detection algorithm in the multi-label classification problem area. In this study, we perform an extensive evaluation of LD3 by comparing it with 14 prevalent supervised concept drift detection algorithms that we adapt to the problem area using 15 datasets and a baseline classifier. The results show that LD3 provides between 16.9 and 56% better predictive performance than comparable detectors on both real-world and synthetic data streams.

In recent years, the data-driven optimization area has seen a shift in the research focus from static batched data environment to dynamic streaming data environment. However, this field is hindered by the lack of a comprehensive and standardized test suite. To fill this gap, we introduce SDDObench, the first benchmark tailored for evaluating and comparing the streaming data-driven evolutionary algorithms (SDDEAs). SDDObench comprises two sets of objective functions combined with five different types of concept drifts, which offer the benefit of being inclusive in generating data streams that mimic various real-world situations, while also facilitating straight-forward description and analysis. As a proof-of-concept study, four well-known algorithms are selected to tackle the problems generated by SDDObench. The experiment results and analysis reveal ongoing challenges in attaining good performance for streaming data-driven optimization. Our SDDObench is open-source and accessible at: https://github.com/LabGong/SDDObench.

One of the most important challenges for machine learning community is to develop efficient classifiers which are able to cope with data streams, especially with the presence of the so-called concept drift. This phenomenon is responsible for the change of classification task characteristics, and poses a challenge for the learning model to adapt itself to the current state of the environment. So there is a strong belief that one-class classification is a promising research direction for data stream analysis--it can be used for binary classification without an access to counterexamples, decomposing a multi-class data stream, outlier detection or novel class recognition. This paper reports a novel modification of weighted one-class support vector machine, adapted to the non-stationary streaming data analysis. Our proposition can deal with the gradual concept drift, as the introduced one-class classifier model can adapt its decision boundary to new, incoming data and additionally employs a forgetting mechanism which boosts the ability of the classifier to follow the model changes. In this work, we propose several different strategies for incremental learning and forgetting, and additionally we evaluate them on the basis of several real data streams. Obtained results confirmed the usability of proposed classifier to the problem of data stream classification with the presence of concept drift. Additionally, implemented forgetting mechanism assures the limited memory consumption, because only quite new and valuable examples should be memorized.

In real-time streaming data, concept drift and class imbalance may occur simultaneously which causes the performance degradation of the online machine learning models. Most of the existing work is limited to addressing these issues for binary class data streams. Very little focus is given to the multi-class data streams. The most common approach to address these issues is ensemble learning. Ensemble learning consists of multiple classifiers combined which are trained on different subsets of the data to improve the overall accuracy. The performance of the ensemble learning approach suffers in case the new classifier is not trained on appropriate data (the data about the new concept). To address this gap, this study has proposed a Smart Adaptive Ensemble Model (SAEM) to address the issues of concept drift and class imbalance for multi-class data streams. The SAEM monitors the feature-level change in data distribution and creates a background ensemble to train the new classifier on features that observe change. To address the class imbalance issue, SAEM applies higher weights on the minority class instances using the dynamic class imbalance ratio. The proposed model outperformed the existing state-of-the-art approaches on the eight different data streams. The results showed an average improvement of 15.857% in accuracy, 20.35% in Kappa, 16.12% in F1-score, 15.58% in precision, and 16.42% in recall. The Friedman test confirmed statistically significant performance differences among all models across five key metrics. Based on the obtained results, the research findings strongly support the notion that SAEM exhibits enhanced effectiveness and efficiency as a solution for online learning applications.

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

Machine learning algorithms have been widely used in IoT application systems for predicting labels of input data streams from IoT sensors. Since data streams are not always stationary in real-world scenarios, underlying data distribution changes may cause a deterioration of the prediction performance that is known as concept drift. Existing concept drift detection methods often assume that complete and true labels for detecting the prediction errors are always available immediately after the prediction. However, such an assumption is not realistic in real IoT application systems. This paper experimentally investigates the robustness of six representative concept drift detectors against unreliable data streams containing the data with error labels and the data with no labels. Robustness is evaluated in terms of the average detection delay and precision by increasing the ratios of error-labeled and missing-label data in a synthetic data stream. Our experimental results show that Cumulative Sum (CUSUM), Page-Hinkley (PH), and Drift Detection Method (DDM) achieve relatively stable performances when the ratio of error-labeled data is less than 40%. With respect to the drift detection efficiency, CUSUM can be regarded as the most robust detector among other detectors tested in our experiments.