A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Anomaly detection is one of the frequent and important subroutines deployed in large-scale data processing applications. Even being a well-studied topic, existing techniques for unsupervised anomaly detection require storing significant amounts of data, which is prohibitive from memory, latency and privacy perspectives, especially for small mobile devices which has ultra-low memory budget and limited computational power. In this paper, we propose ACE (Arrays of (locality-sensitive) Count Estimators) algorithm that can be 60x faster than most state-of-the-art unsupervised anomaly detection algorithms. In addition, ACE has appealing privacy properties. Our experiments show that ACE algorithm has significantly smaller memory footprints (∠ 4MB in our experiments) which can exploit Level 3 cache of any modern processor. At the core of the ACE algorithm, there is a novel statistical estimator which is derived from the sampling view of Locality Sensitive Hashing (LSH). This view is significantly different and efficient than the widely popular view of LSH for near-neighbor search. We show the superiority of ACE algorithm over 11 popular baselines on 3 benchmark datasets, including the KDD-Cup99 data which is the largest available public benchmark comprising of more than half a million entries with ground truth anomaly labels.

Anomaly detection can be seen as an unsupervised learning task in which a predictive model created on historical data is used to detect outlying instances in new data. This work addresses possibly promising but relatively uncommon application of anomaly detection to text data. A Polish Internet discussion forum devoted to psychoactive substances received from home-grown plants, such as hashish or marijuana, serves as a text source that is both realistic and possibly interesting on its own, due to potential associations with drug-related crime. Forum posts are preprocessed by stopword removal, spelling correction, stemming, and frequency-based term filtering. The Global Vectors (GloVe) text representation, which is an example of the increasingly popular word embedding approach, is combined with two unsupervised anomaly detection algorithms, based on one-class SVM classification and based on dissimilarity to k-medoids clusters. The cluster dissimilarity approach combined with the GloVe representation outperforms one-class SVM with respect to detection quality and appears a more promising approach to anomaly detection in text data.

Outlier or anomaly detection is the process through which datum/data with different properties from the rest of the data is/are identified. Their importance lies in their use in various domains such as fraud detection, network intrusion detection, and spam filtering. In this paper, we introduce a new outlier detection algorithm based on an ensemble method and distance-based data filtering with an iterative approach to detect outliers in unlabeled data. The ensemble method is used to cluster the unlabeled data and to filter out potential isolated outliers from the same by iteratively using a cluster membership threshold until the Dunn index score for clustering is maximized. The distance-based data filtering, on the other hand, removes the potential outlier clusters from the post-clustered data based on a distance threshold using the Euclidean distance measure of each data point from the majority cluster as the filtering factor. The performance of our algorithm is evaluated by applying it to 10 real-world machine learning datasets. Finally, we compare the results of our algorithm to various supervised and unsupervised outlier detection algorithms using Precision@n and F-score evaluation metrics.

Nowadays, systems providing user-oriented services often demonstrate periodic patterns due to the repetitive behaviors from people's daily routines. The monitoring data of such systems are time series of observations that record observed system status at sampled times during each day. The periodic feature and multidimensional character of such monitoring data can be well utilized by anomaly detection algorithms to enhance their detection capability. The data periodicity can be used to provide proactive anomaly prediction capability and the correlation among multidimensional series can provide more accurate results than processing the observations separately. However, existing anomaly detection methods only handle one dimensional series and do not consider the data periodicity. In addition, they often require sufficient labelled data to train the models before they can be used. In this paper, we present an unsupervised anomaly detection algorithm called Periodic Self-Organizing Maps (PSOM) to detect anomalies in periodic time series. PSOMs can be used to detect anomalies in multidimensional periodic series as well as one dimensional periodic series and aperiodic series. Our real data evaluation shows that the PSOM outperforms other supervised methods such as SARIMA and Holt-Winters method.

Unsupervised anomaly detection (UAD) attracts a lot of research interest and drives widespread applications, where only anomaly-free samples are available for training. Some UAD applications intend to locate the anomalous regions further even without any anomaly information. Although the absence of anomalous samples and annotations deteriorates the UAD performance, an inconspicuous, yet powerful statistics model, the normalizing flows, is appropriate for anomaly detection (AD) and localization in an unsupervised fashion. The flow-based probabilistic models, only trained on anomaly-free data, can efficiently distinguish unpredictable anomalies by assigning them much lower likelihoods than normal data. Nevertheless, the size variation of unpredictable anomalies introduces another inconvenience to the flow-based methods for high-precision AD and localization. To generalize the anomaly size variation, we propose a novel multiscale flow-based framework (MSFlow) composed of asymmetrical parallel flows followed by a fusion flow to exchange multiscale perceptions. Moreover, different multiscale aggregation strategies are adopted for image-wise AD and pixel-wise anomaly localization according to the discrepancy between them. The proposed MSFlow is evaluated on three AD datasets, significantly outperforming existing methods. Notably, on the challenging MVTec AD benchmark, our MSFlow achieves a new state-of-the-art (SOTA) with a detection AUORC score of up to 99.7%, localization AUCROC score of 98.8% and PRO score of 97.1%.

Supervised deep learning approaches provide state-of-the-art performance on medical image classification tasks for disease screening. However, these methods require large labeled datasets that involve resource-intensive expert annotation. Further, disease screening applications have low prevalence of abnormal samples; this class imbalance makes the task more akin to anomaly detection. While the machine learning community has proposed unsupervised deep learning methods for anomaly detection, they have yet to be characterized on medical images where normal vs. anomaly distinctions may be more subtle and variable. In this work, we characterize existing unsupervised anomaly detection methods on retinal fundus images, and find that they require significant fine tuning and offer unsatisfactory performance. We thus propose an efficient and effective transfer-learning based approach for unsupervised anomaly detection. Our method employs a deep convolutional neural network trained on ImageNet as a feature extractor, and subsequently feeds the learned feature representations into an existing unsupervised anomaly detection method. We show that our approach significantly outperforms baselines on two natural image datasets and two retinal fundus image datasets, all with minimal fine-tuning. We further show the ability to leverage very small numbers of labelled anomalies to improve performance. Our work establishes a strong unsupervised baseline for image-based anomaly detection, alongside a flexible and scalable approach for screening applications.

Anomaly detection has been intensively studied in a variety of research fields, including system and network intrusion detections, fraud detections, etc. Current anomaly detection techniques vastly focus on the detection of the anomalous data. This type of approach could be efficient for the sake of system and network intrusion detection. However, for the social related fraud detection, it is not thorough enough for only applying such approach. One reason is that the ignored social or economic environment can directly affect consumers, who can also be the impersonators. Thereby, in this paper, based on the assumption in microeconomics that every single individual can be treated as a consumer, we propose a novel anomaly detection model via social-economic computing. To the best of our knowledge, this is the pioneer research for anomaly detection in social-economic computing.

Guided wave-based structural health monitoring is extensively utilized in various industrial applications to ensure the integrity of components within industrial systems. Among these monitoring techniques, principal component analysis (PCA) reconstruction methods are widely used for anomaly detection due to their computational efficiency and interoperability. However, existing PCA reconstruction methods are semisupervised anomaly detection approaches that require training on historical normal data and fail to detect anomalous signals within the training set. To address this limitation, this work proposes a correntropy PCA (C-PCA), enabling fully unsupervised anomaly detection on raw training data without requiring label information, when the dataset contains a high proportion of abnormal signals. This method allows anomaly detection on real-time measurements without the need for precleaned historical normal data or can also be used to generate clean data for existing semisupervised anomaly detection methods. In correntropy PCA, principal components are extracted from the correntropy matrix rather than the correlation matrix. The correntropy, representing the statistical dependence between samples of guided waves, is estimated utilizing a Gaussian kernel with a specified kernel width. Through the optimization of the kernel width, the correntropy PCA reconstruction method demonstrates superior anomaly detection performance compared with the standard PCA reconstruction method, especially in scenarios where training data are contaminated by a significant proportion of abnormal signals. Guidelines for the optimization of the kernel width are provided. The effectiveness of the correntropy PCA reconstruction-based anomaly detection method is validated using data collected from ten regions over an 80-day period, encompassing guided waves induced by damage occurring over durations ranging from 2 to 20 days.

Outlier detection or anomaly detection is applied in numerous applications, such as fraud detection, network intrusion detection, manufacturing, and environmental monitoring. Due to the continuous and dynamic characteristics of streaming data, outlier detection over data streams becomes a very challenging task. When analyzing real-time data streams, it is typically impossible to store the entire set of data due to space limitations. Also due to the high data rates, it is necessary to produce the results in a limited amount of time. Parallel processing power of Graphics Processing Units (GPUs) can be used to accelerate the outlier detection process, and thus address the challenges of outlier detection over data streams. This paper proposes a GPU-accelerated outlier detection algorithm for continuous data streams using kernel density estimation approach. Experiments show that the proposed SOD_GPU algorithm is efficient for detecting outliers in high-dimensional, high-speed data streams, and produces results in a timely manner without compromising the outlier detection accuracy. The proposed method achieved up to 20X speedup compared to a respective multi-core CPU implementation, and the speedup increases with the number of data attributes and the input data rate.

To defend a network system from security risks, intrusion detection systems (IDSs) have been playing an important role in recent years. There are two types of detection algorithms of IDSs: misuse detection and anomaly detection. Because misuse detection is based on a signature which is created from the features of attack traffic by security experts, it can achieve accurate and stable detection. However, its weakness is the difficulty of detecting new attacks (i.e., 0-day attack), and the cost of maintaining the latest signature version. Thinking of the increase of the skillful intrusion, e.g., intrusion showing similar access behavior to normal, misuse detection cannot handle these critical attacks, which results in a large number of false alarms. To cope with these problems, we present a clustering algorithm based on an unsupervised anomaly detection. We evaluated our system using Kyoto2006+ data set and KDD Cup 1999 data set. Evaluation results show that our approach achieved a higher detection rate in the region of very low false positive rate and real-time preprocessing capability.

Effectively monitoring the operational status of reactor coolant pumps (RCPs) is crucial for enhancing the safety and stability of nuclear power operations. To address the challenges of limited interpretability and suboptimal detection performance in existing methods for detecting abnormal operating states of RCPs, this paper proposes an interpretable, unsupervised anomaly detection approach. This innovative method designs a framework that combines Kernel Self-Organizing Map (Kernel SOM) clustering with Bayesian Posterior Inference. Specifically, the proposed method uses Kernel SOM to extract typical patterns from normal operation data. Subsequently, a distance probability distribution model reflecting the data distribution structure within each cluster is constructed, providing a robust tool for data distribution analysis for anomaly detection. Finally, based on prior knowledge, such as distance probability distribution, the Bayesian Posterior Inference is employed to infer the probability of the equipment being in a normal state. By constructing distribution models that reflect data distribution structures and combining them with posterior inference, this approach realizes the traceability and interpretability of the anomaly detection process, improving the transparency of anomaly detection and enabling operators to understand the decision logic and the analysis of the causes of anomalous occurrences. Verification via real-world operational data demonstrates the method’s superior effectiveness. This work offers a highly interpretable solution for RCP anomaly detection, with significant implications for safety-critical applications in the nuclear energy sector.

Outliers detection is a task that finds objects that are dissimilar or inconsistent with respect to the remaining data. It has many uses in applications like fraud detection, network intrusion detection and clinical diagnosis of diseases. Using clustering algorithms for outlier detection is a technique that is frequently used. The clustering algorithms consider outlier detection only to the point they do not interfere with the clustering process. This paper compares the performance of the four algorithms on outlier detection efficiency. The main objective is to detect outliers while simultaneously perform clustering operation.

Anomaly detection (AD) aims to distinguish the data points that are inconsistent with the overall pattern of the data. Recently, unsupervised anomaly detection methods have aroused huge attention. Among these methods, feature representation (FR) plays an important role, which can directly affect the performance of anomaly detection. Sparse representation (SR) can be regarded as one of matrix factorization (MF) methods, which is a powerful tool for FR. However, there are some limitations in the original SR. On the one hand, it just learns the shallow feature representations, which leads to the poor performance for anomaly detection. On the other hand, the local geometry structure information of data is ignored. To address these shortcomings, a graph regularized deep sparse representation (GRDSR) approach is proposed for unsupervised anomaly detection in this work. In GRDSR, a deep representation framework is first designed by extending the single layer MF to a multilayer MF for extracting hierarchical structure from the original data. Next, a graph regularization term is introduced to capture the intrinsic local geometric structure information of the original data during the process of FR, making the deep features preserve the neighborhood relationship well. Then, a L1-norm-based sparsity constraint is added to enhance the discriminant ability of the deep features. Finally, a reconstruction error is applied to distinguish anomalies. In order to demonstrate the effectiveness of the proposed approach, we conduct extensive experiments on ten datasets. Compared with the state-of-the-art methods, the proposed approach can achieve the best performance.

Data anomaly detection plays a pivotal role in structural health monitoring (SHM), enabling reliable and accurate assessment of structural condition. This paper proposes an unsupervised anomaly detection method based on the spatio-temporal sparse integration framework (STSIF), which aims to effectively distinguish various types of anomalous patterns in practical applications. First, the anomaly patterns are classified into three-level categories—normal, mild anomaly, and significant anomaly—and then a three-stage unsupervised anomaly detection framework is designed for these categories. At each stage, the characteristics-based detection indicators are proposed to detect significant, mild, and local anomaly progressively. For significant anomaly, the spatial-sparse fusion indicator is proposed and clustered with the unsupervised Gaussian mixture model (GMM). For mild anomaly, the sparse-based reconstruction detection method is proposed for qualitative analysis of noise levels, in which the Pearson correlation between the original and sparse-based reconstructed signal is computed for the GMM clustering. Overall accuracy of 98.36% and 95.51% is obtained for the data sets on two real-world bridges, respectively. Finally, regarding local anomaly detection, the proposed mask-based ResNet18 effectively emphasizes the local contextual information within the signal and substantially reduces the risk of overgeneralization. In summary, the proposed STSIF could efficiently detect various anomaly patterns without substantial training data or complex parameter tuning, providing a comprehensive detection paradigm for SHM.

An increase in the collection of physiological signals, whether done implicitly in wearable or IoT device or explicitly in experimental and laboratory environments, creates the need for development of smart systems and tools capable of data analysis with limited expert knowledge. Anomaly detection, specifically unsupervised anomaly detection can be used to design a tool or even as a tool to help remedy this issue. This paper will focus on how unsupervised anomaly detection can be utilized for the development of such systems. A systematic, robust, and customizable approach will be presented and preliminary results will be shown that open the door for future research and algorithm development.