VOA*: Fast Angle-Based Outlier Detection over High-Dimensional Data Streams

The rapid growth of data streams, propelled by the proliferation of sensors and Internet of Things (IoT) devices, presents significant challenges for real-time clustering of high-dimensional data. Traditional clustering algorithms struggle with high dimensionality, memory and time constraints, and adapting to dynamically evolving data. Existing dimensionality reduction methods often neglect feature ranking, leading to suboptimal clustering performance. To address these issues, we introduce E-Stream, a novel entropy-based clustering algorithm for high-dimensional data streams. E-Stream performs real-time feature ranking based on entropy within a sliding time window to identify the most informative features, which are then utilized with the DenStream algorithm for efficient clustering. We evaluated E-Stream using the NSL-KDD dataset, comparing it against DenStream, CluStream, and MR-Stream. The evaluation metrics included the average F-Measure, Jaccard Index, Fowlkes-Mallows Index, Purity, and Rand Index. The results show that E-Stream outperformed the baseline algorithms in both clustering accuracy and computational efficiency while effectively reducing dimensionality. E-Stream also demonstrated significantly less memory consumption and fewer computational requirements, highlighting its suitability for real-time processing of high-dimensional data streams. Despite its strengths, E-Stream requires manual parameter adjustment and assumes a consistent number of active features, which may limit its adaptability to diverse datasets. Future work will focus on developing a fully autonomous, parameter-free version of the algorithm, incorporating mechanisms to handle missing features and improving the management of evolving clusters to enhance robustness and adaptability in dynamic IoT environments.

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Merit-guided dynamic feature selection filter for data streams

PTS: Projected Topological Stream clustering algorithm

Concept drift, the phenomenon where the underlying data distribution changes over time, presents significant challenges for machine learning models deployed in realtime applications. This issue is particularly pronounced in high-dimensional data streams, where the complexity of monitoring and adapting to these changes can lead to decreased model performance and reliability. This literature review examines existing research on concept drift in high-dimensional data environments, exploring the causes, detection methods and techniques for adaptation. We delve into classical statistical approaches, machine learning and deep learning-based methods, and discuss the inherent challenges posed by highdimensionality. Additionally, in this paper highlight evaluation metrics, benchmark datasets, and a comparative analysis of the strengths and weaknesses of current techniques. This paper concludes with potential future research directions, emphasizing the importance of scalable, adaptive and hybrid approaches to tackle concept drift effectively in high-dimensional data streams.

Change-point detection is of great interest in applications such as target tracking, anomaly detection and trend filtering. In many cases, it is also desirable to localize the change- point, if it exists. Motivated by the unprecedented scale and rate of modern high-dimensional streaming data, we propose a change-point detection and estimation procedure based on data sketching, which only requires a single sketch per high- dimensional data vector, by cyclically applying a small set of Gaussian sketching vectors. We demonstrate that when the underlying changes exhibit certain low-dimensional structures, such as sparsity, and the signal-to-noise ratio is not too small, the change-points can be reliably detected and located with a small number of sketching vectors based on filtering via convex optimization. Our procedure can be implemented in an online fashion to handle multiple change-points, since it sequentially operates on small windows of observations.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

Outlier detection is of great significance in the domain of data mining. Its task is to find those target points that are not identical to most of the object generation mechanisms. The existing algorithms are mainly divided into density-based algorithms and distance-based algorithms. However, both approaches have some drawbacks. The former struggles to handle low-density modes, while the latter cannot detect local outliers. Moreover, the outlier detection algorithm is very sensitive to parameter settings. This paper proposes a new two-parameter outlier detection (TPOD) algorithm. The method proposed in this paper does not need to manually define the number of neighbors, and the introduction of relative distance can also solve the problem of low density and further accurately detect outliers. This is a combinatorial optimization problem. Firstly, the number of natural neighbors is iteratively calculated, and then the local density of the target object is calculated by adaptive kernel density estimation. Secondly, the relative distance of the target points is computed through natural neighbors. Finally, these two parameters are combined to obtain the outlier factor. This eliminates the influence of parameters that require users to determine the number of outliers themselves, namely, the top-n effect. Two synthetic datasets and 17 real datasets were used to test the effectiveness of this method; a comparison with another five algorithms is also provided. The AUC value and F1 score on multiple datasets are higher than other algorithms, indicating that outliers can be found accurately, which proves that the algorithm is effective.

Motivated by applications to root-cause identification of faults in high-dimensional data streams that may have very limited samples after faults are detected, we consider multiple testing in models for multivariate statistical process control (SPC). With quick fault detection, only small portion of data streams being out-of-control (OC) can be assumed. It is a long standing problem to identify those OC data streams while controlling the number of false discoveries. It is challenging due to the limited number of OC samples after the termination of the process when faults are detected. Although several false discovery rate (FDR) controlling methods have been proposed, people may prefer other methods for quick detection. With a recently developed method called Knockoff filtering, we propose a knockoff procedure that can combine with other fault detection methods in the sense that the knockoff procedure does not change the stopping time, but may identify another set of faults to control FDR. A theorem for the FDR control of the proposed procedure is provided. Simulation studies show that the proposed procedure can control FDR while maintaining high power. We also illustrate the performance in an application to semiconductor manufacturing processes that motivated this development.

Stream data classification is a research topic of growing interest. Traditional approaches treat all attributes (i.e. features) of a stream data object fairly during the process of classification. Yet, in a real streaming environment, not all of the features are equally important to the classification result. Therefore, the classification accuracy can be improved by highlighting representative features and dimming irrelevant features. We apply a feature weighted ensemble classification model to solve this problem. The model is built using a modified K-means clustering technique and classification is performed with K-nearest neighbor algorithm. Experiments show that the method can improve the accuracy of classification, especially when there are noise features in high-dimensional stream data.

Outlier detection is a key technique in data ming and machine learning fields. The deviating characters of outliers make huge detrimental effects on the learning tasks. A lot of algorithms are therefore proposed to handle outliers from different perspectives, such as distance, density, angle and so on. Among these approaches, the density-based methods achieve better performance, but also suffer from huge time complexity. Recently, in order to accelerate the speed and improve the performance, the subsampling ensemble method attracts much attention, which has a reasonable theoretical interpretation and high performance. However, existing work only gives the partial picture of outlier detection via row-sampling, the effective portfolio of bi-sampling is still void. In light of this, we propose the general outlier detection framework via bi-sampling, Bi-Sampling Outlier Detection (BSOD) and provide the effective portfolios of the row and column-sampling ratios in a theoretical way. In addition, the benefits of BSOD are fully illustrated in terms of ensemble diversity and divide-and-conquer. Further we employ LOF within BSOD as BI-LOF to conduct extensive experiments. In general, on 30 synthetic and 17 real-world data sets we thoroughly explore the characteristics of BI-LOF with different numbers of instances, features, nearest neighbors, validate the theoretical analysis of BSOD condition on synthetic data sets, and show obvious advantages over other state-of-the-art algorithms in terms of low and high dimensional real-world data sets. And finally we use BI-LOF to conduct image outlier detection and show high quality and stableness of BI-LOF.

Due to the demand for tackling the problem of streaming data with high-dimensional covarites, we propose an online sparse sliced inverse regression (OSSIR) method for online sufficient dimension reduction (SDR). The existing online SDR methods focus on the case when [Formula: see text] (dimension of covariates) is small. In this paper, we adapt the sparse sliced inverse regression to cope with high-dimensional streaming data where the dimension [Formula: see text] is large. There are two important steps in our method, one is to extend the online principal component analysis to iteratively obtain the eigenvalues and eigenvectors of the kernel matrix, the other is to use the truncated gradient to perform online [Formula: see text] regularization. Theoretical properties of the proposed online learner are established. By comparing with several existing methods in simulations and real data applications, we demonstrate the effectiveness and efficiency of our method.

Anomaly detection is a cornerstone of modern intelligent systems, with critical applications in healthcare, cybersecurity, smart grids, and IoT environments. While traditional machine learning and deep learning models have shown promise in identifying outliers, they often face challenges such as dependence on large labeled datasets, high computational costs, and limited scalability to edge devices or high-dimensional data streams. This study introduces D2H-AD, a novel anomaly detection framework built upon Hyperdimensional Computing (HDC), a brain-inspired paradigm that encodes information using high-dimensional distributed vectors. Unlike prior HDC-based approaches, D2H-AD fuses distance-based similarity and density-aware encoding in a hybrid design, significantly improving anomaly characterization and detection accuracy. Ablation experiments confirm that hyperdimensional encoding alone contributes up to 5.4% higher ROC-AUC compared to applying the same density-distance scoring in the original Euclidean feature space, and D2H-AD consistently outperforms all five published baselines (HDAD, ODHD, One-Class SVM, Isolation Forest, and Autoencoders) across every evaluated dataset. The method is designed to be lightweight, interpretable, and computationally efficient, suggesting strong suitability for deployment in resource-constrained and real-time environments. To validate its effectiveness, we evaluate D2H-AD on five diverse benchmark datasets, comparing its performance against five leading techniques: HDAD, ODHD, One-Class SVM, Isolation Forest, and Autoencoders. Our results show that D2H-AD consistently achieves superior F1 scores and ROC-AUC metrics, demonstrating robustness against class imbalance, noise, and data complexity. Beyond accuracy, D2H-AD offers practical advantages including scalability, minimal memory footprint, and an expected low-latency profile derived from its binary operations and lightweight design. These characteristics are crucial for TinyML and edge AI applications. This work highlights the untapped potential of HDC for high-performance anomaly detection and opens new avenues for secure, interpretable, and energy-efficient AI solutions in dynamic environments such as IoT, embedded systems, and beyond. Additionally, D2H-AD provides feature-level interpretability through hypervector decoding, enabling transparent explanations for safety-critical applications.

High-dimensional data streams pose challenges to traditional clustering algorithm, due to their inherent sparsity and data tend to cluster in different subspaces of the entire feature space. In this paper, we resolve the subspace clustering problem by mining correlated-clusters, in which selected features are correlated with each other. Moreover, taking data evolution in data streams into account, we propose methods to mine correlations of features incrementally and adaptively. At each time tick t, according to our proposed multiple regression measurement, we cluster the newly arrived data sample to one of correlated-clusters whose local correlations fit to the data sample and also update the local correlations adaptively, based on an incremental principal component analysis technology. The results of experiments on high-dimensional synthetic data and real data demonstrate that our methods can achieve higher accuracy of query than related work and perform much more efficiently. Additionally, our proposed methods are able to forecast missing values in streaming data successfully.

Recently, many Intrusion Detection Systems (IDS) have been proposed to detect and report about malicious nodes in Mobile Ad Hoc Networks (MANETs) by using various outlier detection algorithms. All these algorithms have limitations in terms of accuracy and speed. In this paper, we propose a new outlier detection algorithm called Weighted Distance Based Outlier Detection (WDBOD) algorithm to detect the intruders in wireless environment. The experimental results show that the proposed algorithm improves the detection accuracy and reduces the false alarm rate.