COPOD: Copula-Based Outlier Detection

Outlier detection shows its increasingly high practical value in many application areas such as intrusion detection, fraud detection, discovery of criminal activities in electronic commerce and so on. Many techniques have been developed for outlier detection, including distribution-based outlier detection algorithm, depth-based outlier detection algorithm, distance-based outlier detection algorithm, density-based outlier detection algorithm and clustering-based outlier detection. Spectral clustering receives much attention as a competitive clustering algorithms emerging in recent years. However, it is not very well scalable to modern large datasets. To partially circumvent this drawback, in this paper, we propose a new outlier detection method inspired by spectral clustering. Our algorithm combines the concept of kNN and spectral clustering techniques to obtain the abnormal data as outliers by using the information of eigenvalues and eigenvectors statistically in the feature space. We compare the performance of our methods with distance-based outlier detection methods and density-based outlier detection methods. Experimental results show the effectiveness of our algorithm for identifying outliers.

Outlier detection is of great significance in the domain of data mining. Its task is to find those target points that are not identical to most of the object generation mechanisms. The existing algorithms are mainly divided into density-based algorithms and distance-based algorithms. However, both approaches have some drawbacks. The former struggles to handle low-density modes, while the latter cannot detect local outliers. Moreover, the outlier detection algorithm is very sensitive to parameter settings. This paper proposes a new two-parameter outlier detection (TPOD) algorithm. The method proposed in this paper does not need to manually define the number of neighbors, and the introduction of relative distance can also solve the problem of low density and further accurately detect outliers. This is a combinatorial optimization problem. Firstly, the number of natural neighbors is iteratively calculated, and then the local density of the target object is calculated by adaptive kernel density estimation. Secondly, the relative distance of the target points is computed through natural neighbors. Finally, these two parameters are combined to obtain the outlier factor. This eliminates the influence of parameters that require users to determine the number of outliers themselves, namely, the top-n effect. Two synthetic datasets and 17 real datasets were used to test the effectiveness of this method; a comparison with another five algorithms is also provided. The AUC value and F1 score on multiple datasets are higher than other algorithms, indicating that outliers can be found accurately, which proves that the algorithm is effective.

Recently, many Intrusion Detection Systems (IDS) have been proposed to detect and report about malicious nodes in Mobile Ad Hoc Networks (MANETs) by using various outlier detection algorithms. All these algorithms have limitations in terms of accuracy and speed. In this paper, we propose a new outlier detection algorithm called Weighted Distance Based Outlier Detection (WDBOD) algorithm to detect the intruders in wireless environment. The experimental results show that the proposed algorithm improves the detection accuracy and reduces the false alarm rate.

In order to solve the problem that the existing outlier detection algorithm is difficult to detect the one-dimensional integer data set with uneven frequency distribution and uniform distance distribution and low accuracy, the advantages of density outlier detection and distance outlier detection can be combined. An outlier detection algorithm DAD (Density and Distance) based on density and distance was proposed. In order to improve the possibility of outlier sample distance, the algorithm can define the weight distance; introduce the global density outlier factor combined with the weight distance as the relative distance, and use the cutting edge strategy to quickly cut the outliers based on the minimum spanning tree. Then, an artificial data set was used to test the algorithm. Experimental results showed that the algorithm had a good outlier detection accuracy in dealing with data sets with uneven frequency distribution and uniform distance distribution.

Outlier detection is a well-researched and crucial problem in machine learning. However, there is little research on string data outlier detection, as most literature focuses on outlier detection of numerical data. A robust string data outlier detection algorithm could assist with data cleaning or anomaly detection in system log files. In this thesis, we compare two string outlier detection algorithms. Firstly, we introduce a variant of the well-known local outlier factor algorithm, which we tailor to detect outliers on string data using the Levenshtein measure to calculate the density of the dataset. We present a differently weighted Levenshtein measure, which considers hierarchical character classes and can be used to tune the algorithm to a specific string dataset. Secondly, we introduce a new kind of outlier detection algorithm based on the hierarchical left regular expression learner, which infers a regular expression for the expected data. Using various datasets and parameters, we experimentally show that both algorithms can conceptually find outliers in string data. We show that the regular expression-based algorithm is especially good at finding outliers if the expected values have a distinct structure that is sufficiently different from the structure of the outliers. In contrast, the local outlier factor algorithms are best at finding outliers if their edit distance to the expected data is sufficiently distinct from the edit distance between the expected data.

Unsupervised outlier detection constitutes a crucial phase within data analysis and remains an open area of research. A good outlier detection algorithm should be computationally efficient, robust to tuning parameter selection, and perform consistently well across diverse underlying data distributions. We introduce Boundary Peeling, an unsupervised outlier detection algorithm. Boundary Peeling uses the average signed distance from iteratively peeled, flexible boundaries generated by one-class support vector machines to flag outliers. The method is similar to convex hull peeling but well suited for high-dimensional data and has flexibility to adapt to different distributions. Boundary Peeling has robust hyperparameter settings and, for increased flexibility, can be cast as an ensemble method. In unimodal and multimodal synthetic data simulations Boundary Peeling outperforms all state of the art methods when no outliers are present while maintaining comparable or superior performance in the presence of outliers. Boundary Peeling performs competitively or better in terms of correct classification, AUC, and processing time using semantically meaningful benchmark datasets.

The satellite missions CHAMP, GRACE, and GOCE mark the beginning of a new era in gravity field determination and modeling. They provide unique models of the global stationary gravity field and its variation in time. Due to inevitable measurement errors, sophisticated pre-processing steps have to be applied before further use of the satellite measurements. In the framework of the GOCE mission, this includes outlier detection, absolute calibration and validation of the SGG (satellite gravity gradiometry) measurements, and removal of temporal effects. In general, outliers are defined as observations that appear to be inconsistent with the remainder of the data set. One goal is to evaluate the effect of additive, innovative and bulk outliers on the estimates of the spherical harmonic coefficients. It can be shown that even a small number of undetected outliers (<0.2 of all data points) can have an adverse effect on the coefficient estimates. Consequently, concepts for the identification and removal of outliers have to be developed. Novel outlier detection algorithms are derived and statistical methods are presented that may be used for this purpose. The methods aim at high outlier identification rates as well as small failure rates. A combined algorithm, based on wavelets and a statistical method, shows best performance with an identification rate of about 99%. To further reduce the influence of undetected outliers, an outlier detection algorithm is implemented inside the gravity field solver (the Quick-Look Gravity Field Analysis tool was used). This results in spherical harmonic coefficient estimates that are of similar quality to those obtained without outliers in the input data.

Outlier detection techniques are well known for identifying a small amount of data objects named outliers that are far away from clusters and exist in sparse regions of data space. However, most outlier detection algorithms based on k nearest neighbors are sensitive to parameter k. The outlier detection algorithms based on clustering rely on specific clustering algorithms, and outliers are by-products. To partially circumvent these problems, motivated by the medoid concept of K-medoids clustering algorithms, in this paper, we propose a k-nearest neighbor medoid-based outlier detection method that is easy to implement and can provide competing performances with existing solutions. At the same time, a method to determine the parameter k is proposed in combination with the outlier detection algorithm proposed in this paper. Experiments performed on real datasets demonstrate the efficacy of our method.

Recently, outlier detection has widespread applications in different areas. The task is to identify outliers in the dataset and extract potential information. The existing outlier detection algorithms mainly do not solve the problems of parameter selection and high computational cost, which leaves enough room for further improvements. To solve the above problems, our paper proposes a parameter-free outlier detection algorithm based on dataset optimization method. Firstly, we propose a dataset optimization method (DOM), which initializes the original dataset in which density is greater than a specific threshold. In this method, we propose the concepts of partition function (P) and threshold function (T). Secondly, we establish a parameter-free outlier detection method. Similarly, we propose the concept of the number of residual neighbors, as the number of residual neighbors and the size of data clusters are used as the basis of outlier detection to obtain a more accurate outlier set. Finally, extensive experiments are carried out on a variety of datasets and experimental results show that our method performs well in terms of the efficiency of outlier detection and time complexity.

An outlier detection algorithm for categorical matrix-object data

Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis

We investigate the use of biased sampling according to the density of the data set to speed up the operation of general data mining tasks, such as clustering and outlier detection in large multidimensional data sets. In density-biased sampling, the probability that a given point will be included in the sample depends on the local density of the data set. We propose a general technique for density-biased sampling that can factor in user requirements to sample for properties of interest and can be tuned for specific data mining tasks. This allows great flexibility and improved accuracy of the results over simple random sampling. We describe our approach in detail, we analytically evaluate it, and show how it can be optimized for approximate clustering and outlier detection. Finally, we present a thorough experimental evaluation of the proposed method, applying density-biased sampling on real and synthetic data sets, and employing clustering and outlier detection algorithms, thus highlighting the utility of our approach.

The goal of outlier detection is to pinpoint data points that exhibit notable deviations from the rest of the observed values. It has found successful application in numerous fields, including process inspection, anti-terrorist operations, and public security. However, the majority of existing outlier algorithms rely on methods involving filling or deleting missing data, with few directly addressing incomplete data. This article studies outlier detection for incomplete real-valued data based on class-consistent technology, rough set theory, and granular computing. First, a tolerance relation founded on class-consistent technology is presented to illustrate the similarity among information values within an incomplete real-valued information system (IRVIS). Then, the tolerance classes are established based on the tolerance relation and utilized for computing approximate accuracy and other metrics. Next, an outlier factor is defined, considering both the degree of outlierness and the weight function assigned to each object within an IRVIS, elucidating the uncertainty and degree of outlier. Finally, an outlier detection algorithm (ODIRG) for an IRVIS based on class-consistent technology, rough set theory, and granular computing is devised. Numerical experiments on seven UCI datasets are undertaken to evaluate the stability of the ODIRG algorithm. The proposed method is demonstrated to exhibit strong effectiveness and adaptability for categorical data when compared with five other algorithms. It is notable that for comprehensive comparison, precision, recall, F F 1-measure, and receiver operating characteristic curve are employed to delineate the benefits of the proposed approach.

Outlier detection is one of the major data mining methods. This paper proposes a three-step approach to detect spatio-temporal outliers in large databases. These steps are clustering, checking spatial neighbors, and checking temporal neighbors. In this paper, we introduce a new outlier detection algorithm to find small groups of data objects that are exceptional when compared with rest large amount of data. In contrast to the existing outlier detection algorithms, new algorithm has the ability of discovering outliers according to the non-spatial, spatial and temporal values of the objects. In order to demonstrate the new algorithm, this paper also presents an example application using a data warehouse

Outlier Detection is a key technique to discover abnormal patterns from mass data. Pivot based metric space outlier detection algorithm is designed to improve the cutoff value of outlier degree faster and terminate the detection program earlier, for saving detection time. However, due to the lack of efficient pivot selection method, the performance of existing related algorithms is not as good as it might have been. In this paper, we propose an adaptive cutoff distance based density peak pivot selection algorithm (ADP) to get a suitable pivot quickly. Moreover, we also develop an improved outlier detection algorithm based on ADP algorithm (ADPOD). Experimental results indicate that ADPOD can effectively overcome the existing relate algorithm, and achieves a 2.67 speed up over it on average and, in certain cases, up to 8.06. The distance calculation times are reduced by 53.39% on average and up to 94.84%, as well as acceptable indexing time and the universality of metric space.