Outlier detection via sampling ensemble

Regarding the magnificent ability of the fuzzy logic in explanation and analyzing the real phenomena, using fuzzy data is growing rapidly. But the tools for doing data mining in a fuzzy data set; especially in high dimension, are not sufficient. Considering the remarkable ability of the LoOP (Local Outlier Probabilities) method in outlier detecting in high dimensional crisp data set, like supplying for each data object, determining outlier probability as scar which is easily explainable competitor on various synthetic and real world data sets, the aim of this article is to present an extended version of this method for using in high dimensional fuzzy database. In order to provide a completely fuzzy strategy, first we propose a new approach to assigning distance between fuzzy high dimension data based on Hausdorff and Vertex distance. After proving that this distance measuring is a metric, since LoOP is a distance based method, we redefined the LoOP algorithm step by step based on the new meter. Then we allocate an outlier score in the range of [0,1] to each data based on LoOP idea, to identify the outliers. The presented numerical tests show the efficiency of the new method, for random cases of knowing and unknowing the outlier and a real case. The results shows also the stability and robustness of the presented method more that its accuracy.

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Defining a boundary between inliers and outliers is a major challenge in unsupervised outlier detection. In the absence of labeled data, the true outliers set cannot be evaluated. This lays the burden on both the choice of an efficient outlier detection criterion, and parameter selection. While numerous unsupervised outlier detection criteria, with different parameters, have been proposed, an unsupervised evaluation of outliers is still missing. This work introduces a theoretical basis, and proposes a validity index, to evaluate the quality of outliers. This is not a trivial problem when nothing is known about the structure and density of the data. The proposed index considers the outlierness quality, the deviation between characteristics of outliers and inliers, and the data distortion. Low and high dimensional data sets are used to evaluate the proposed index.

A novel clustering algorithm CMune is presented for the purpose of finding clusters of arbitrary shapes, sizes and densities in high dimensional feature spaces. It can be considered as a variation of the Shared Nearest Neighbor algorithm (SNN), in which each sample data point votes for the points in its k-nearest neighborhood. Sets of points sharing a common mutual nearest neighbor are considered as dense regions/blocks. These blocks are the seeds from which clusters may grow up. Therefore, CMune is not a point-to-point clustering algorithm. Rather, it is a block-to-block clustering technique. Much of its advantages come from this fact: Noise points and outliers correspond to blocks of small sizes, and homogeneous blocks highly overlap. The algorithm has been applied to a variety of low and high dimensional data sets with superior results over existing techniques such as K-means, DBScan, Mitosis and Spectral clustering. The quality of its results as well as its time complexity, place it at the front of these techniques.

CANF: Clustering and anomaly detection method using nearest and farthest neighbor

Outlier detection in the high-dimensional data stream is a challenging data mining task. In high-dimensional data, the distance-based measures of outlierness become less effective and unreliable. Angle-based outlier detection ABOD technique was proposed as a more suitable scheme for high-dimensional data. However, ABOD is designed for static datasets and its naive application on a sliding window over data streams will result in poor performance. In this research, we propose two incremental algorithms for fast outlier detection based on an outlier threshold value in high-dimensional data streams: IncrementalVOA and \(VOA^{*}\). IncrementalVOA is a basic incremental algorithm for computing outlier factor of each data point in each window. \(VOA^{*}\) enhances the incremental computation by using a bound-based pruning method and a retrospect-based incremental computation technique. The effectiveness and efficiency of the proposed algorithms are experimentally evaluated on synthetic and real world datasets where \(VOA^{*}\) outperformed other methods.

There is no individual classification technique has been shown to deal with all kinds of classification problems. The objective is to select the technique which more possibly reaches the best performance for any domain of data set. We focus on classifying datasets in different domains and properties such as numerical, categorical, and textual. We deal with one versus all strategy to handle multi-class problems. In the experiment, we compared the performance of 4 classification techniques namely Boosted C5.0, KNN, Naïve Bayes, and SVM on 10-fold cross-validation on different number of features. For numerical data set (low and high dimensional data set), the performance of KNN was better than other classification methods. For a categorical and textual data set, Naïve Bayes and SVM were outperformed, respectively.

Anomaly detection in streaming data is an important task for many real-world applications, such as network security, fraud detection, and system monitoring. However, streaming data often exhibit concept drift, which means that the data distribution changes over time. This poses a significant challenge for many anomaly detection algorithms, as they need to adapt to the evolving data to maintain high detection accuracy. Existing streaming anomaly detection algorithms lack a unified evaluation framework that validly assesses their performance and robustness under different types of concept drifts and anomalies. In this paper, we conduct a systematic technical review of the state-of-the-art methods for anomaly detection in streaming data. We propose a new data generator, called SCAR (Streaming data generator with Customizable Anomalies and concept dRifts), that can synthesize streaming data based on synthetic and real-world datasets from different domains. Furthermore, we adapt four static anomaly detection models to the streaming setting using a generic reconstruction strategy as baselines, and then compare them systematically with 9 existing streaming anomaly detection algorithms on 76 synthesized datasets that have various types of anomalies and concept drifts. The challenges and future research directions for anomaly detection in streaming data are also presented.

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

Anomaly detection techniques are widely used in a variety of applications, e.g., computer networks, security systems, etc. This paper describes and analyzes an approach to anomaly detection using proximity graphs and the PageRank algorithm. We run a variant of the PageRank algorithm on top of a proximity graph comprised of data points as vertices, which produces a score quantifying the extent to which each data point is anomalous. Previous work in this direction requires first forming a density estimate using the training data, e.g., using kernel methods, and this step is very computationally intensive for high-dimensional data sets. Under mild assumptions and appropriately chosen parameters, we show that PageRank produces point-wise consistent probability density estimates for the data points in an asymptotic sense, and with much less computational effort. As a result, big improvements in terms of running time are witnessed while maintaining similar detection performance. Experiments with synthetic and real-world data sets illustrate that the proposed approach is computationally tractable and scales well to large high-dimensional data sets.

<em>This paper presents a proposed supervised classification technique namely flexible partial histogram Bayes (fPHBayes) learning algorithm. In our previous work, partial histogram Bayes (PHBayes) learning algorithm showed some advantages in the aspects of speed and accuracy in classification tasks. However, its accuracy declines when dealing with small number of instances or when the class feature distributes in wide area. In this work, the proposed fPHBayes solves these limitations in order to increase the classification accuracy. fPHBayes was analyzed and compared with PHBayes and other standard learning algorithms like first nearest neighbor, nearest subclass mean, nearest class mean, naive Bayes and Gaussian mixture model classifier. The experiments were performed using both real data and synthetic data considering different number of instances and different variances of Gaussians. The results showed that fPHBayes is more accurate and flexible to deal with different number of instances and different variances of Gaussians as compared to PHBayes.</em>

Unsupervised anomaly detection algorithm is typically suitable only to a specific type of anomaly, among point anomaly, collective anomaly, and contextual anomaly. A mismatch between the intended anomaly type of an algorithm and the actual type in the data can lead to poor performance. In this paper, utilizing Hilbert–Schmidt independence criterion (HSIC), we propose an unsupervised backward elimination feature selection algorithm BAHSIC-AD to identify a subset of features with the strongest interdependence for anomaly detection. Using BAHSIC-AD, we compare the effectiveness of a recent Spectral Ranking for Anomalies (SRA) algorithm with other popular anomaly detection methods on a few synthetic datasets and real-world datasets. Furthermore, we demonstrate that SRA, combined with BAHSIC-AD, can be a generally applicable method for detecting point, collective, and contextual anomalies.

The explosion of modeling complex systems using attributed networks boosts the research on anomaly detection in such networks, which can be applied in various high-impact domains. Many existing attempts, however, do not seriously tackle the inherent multi-view property in attribute space but concatenate multiple views into a single feature vector, which inevitably ignores the incompatibility between heterogeneous views caused by their own statistical properties. Actually, the distinct but complementary information brought by multi-view data promises the potential for more effective anomaly detection than the efforts only based on single-view data. Furthermore, the abnormal patterns naturally behave diversely in different views, which coincides with people’s desire to discover specific abnormality according to their preferences for views (attributes). Most existing methods cannot adapt to people’s requirements as they fail to consider the idiosyncrasy of user preferences. Therefore, we propose a multi-view framework <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Alarm</small> to incorporate user preferences into anomaly detection and simultaneously tackle heterogeneous attribute characteristics through multiple graph encoders and a well-designed aggregator that supports self-learning and user-guided learning. Experiments on synthetic and real-world datasets, e.g., Disney, Books, and Enron, corroborate the improvement of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Alarm</small> in detection accuracy evaluated by the AUC metric and its effectiveness in supporting user-oriented anomaly detection.

Anomaly detection is valuable in many domains. Distance-based algorithms are one of the most well-known techniques for anomaly detection. These algorithms have been shown to produce good accuracy in a wide range of applications. However, distance-based algorithms are limited in terms of scalability. This makes it difficult for these algorithms to handle massive data sets. To overcome this limitation, in this paper we propose a parallel formulation of the popular KNN distance-based algorithm for unsupervised anomaly detection. Our formulation focused on improving scalability by analyzing in parallel different parts of the data. This approach is valuable whenever the analyzed data does not fit comfortably into memory, or when it is distributed across several locations. To reduce synchronization latency, the proposed algorithm computes approximate nearest neighborhoods. We show, both theoretically and empirically, how this approach greatly improves scalability without significantly penalizing detection accuracy. To validate our approach, we perform extensive experiments on both synthetic and real-world data sets.

Network Intrusion Detection Systems (NIDSs) are an increasingly important tool for the prevention and mitigation of cyber attacks. Over the past years, a lot of research efforts have aimed at leveraging the increasingly powerful models of Machine Learning (ML) for this purpose. A number of labelled synthetic datasets have been generated and made publicly available by researchers, and they have become the benchmarks via which new ML-based NIDS classifiers are being evaluated. Recently published results show excellent classification performance with these datasets, increasingly approaching 100 percent performance across key evaluation metrics such as Accuracy, F1 score, AUC, etc. Unfortunately, we have not yet seen these excellent academic research results translated into practical NIDS systems with such near-perfect performance. This motivated our research presented in this paper, where we analyse the statistical properties of the benign traffic in three of the more recent and relevant NIDS datasets, (CIC_IDS, UNSW_NB15, TON_IOT), by converting them into a common flow format. As a comparison, we consider two datasets obtained from real-world production networks, one from a university network and one from a medium size Internet Service Provider (ISP). Our results show that the two real-world datasets are quite similar among themselves in regards to most of the considered statistical features. Equally, the three synthetic datasets are also relatively similar within their group. However, and most importantly, our results show a distinct difference of most of the considered statistical features between the three synthetic datasets and the two real-world datasets. Since ML relies on the basic assumption of training and test datasets being sampled from the same distribution, this raises the question of how well the performance results of ML-classifiers trained on the considered synthetic datasets can translate and generalise to real-world networks. We believe this is an interesting and relevant question which provides motivation for further research in this space.