Density-Distance Outlier Detection Algorithm Based on Natural Neighborhood

Outlier detection shows its increasingly high practical value in many application areas such as intrusion detection, fraud detection, discovery of criminal activities in electronic commerce and so on. Many techniques have been developed for outlier detection, including distribution-based outlier detection algorithm, depth-based outlier detection algorithm, distance-based outlier detection algorithm, density-based outlier detection algorithm and clustering-based outlier detection. Spectral clustering receives much attention as a competitive clustering algorithms emerging in recent years. However, it is not very well scalable to modern large datasets. To partially circumvent this drawback, in this paper, we propose a new outlier detection method inspired by spectral clustering. Our algorithm combines the concept of kNN and spectral clustering techniques to obtain the abnormal data as outliers by using the information of eigenvalues and eigenvectors statistically in the feature space. We compare the performance of our methods with distance-based outlier detection methods and density-based outlier detection methods. Experimental results show the effectiveness of our algorithm for identifying outliers.

Recently, many Intrusion Detection Systems (IDS) have been proposed to detect and report about malicious nodes in Mobile Ad Hoc Networks (MANETs) by using various outlier detection algorithms. All these algorithms have limitations in terms of accuracy and speed. In this paper, we propose a new outlier detection algorithm called Weighted Distance Based Outlier Detection (WDBOD) algorithm to detect the intruders in wireless environment. The experimental results show that the proposed algorithm improves the detection accuracy and reduces the false alarm rate.

In order to solve the problem that the existing outlier detection algorithm is difficult to detect the one-dimensional integer data set with uneven frequency distribution and uniform distance distribution and low accuracy, the advantages of density outlier detection and distance outlier detection can be combined. An outlier detection algorithm DAD (Density and Distance) based on density and distance was proposed. In order to improve the possibility of outlier sample distance, the algorithm can define the weight distance; introduce the global density outlier factor combined with the weight distance as the relative distance, and use the cutting edge strategy to quickly cut the outliers based on the minimum spanning tree. Then, an artificial data set was used to test the algorithm. Experimental results showed that the algorithm had a good outlier detection accuracy in dealing with data sets with uneven frequency distribution and uniform distance distribution.

Recently, outlier detection has widespread applications in different areas. The task is to identify outliers in the dataset and extract potential information. The existing outlier detection algorithms mainly do not solve the problems of parameter selection and high computational cost, which leaves enough room for further improvements. To solve the above problems, our paper proposes a parameter-free outlier detection algorithm based on dataset optimization method. Firstly, we propose a dataset optimization method (DOM), which initializes the original dataset in which density is greater than a specific threshold. In this method, we propose the concepts of partition function (P) and threshold function (T). Secondly, we establish a parameter-free outlier detection method. Similarly, we propose the concept of the number of residual neighbors, as the number of residual neighbors and the size of data clusters are used as the basis of outlier detection to obtain a more accurate outlier set. Finally, extensive experiments are carried out on a variety of datasets and experimental results show that our method performs well in terms of the efficiency of outlier detection and time complexity.

An outlier detection algorithm for categorical matrix-object data

We investigate the use of biased sampling according to the density of the data set to speed up the operation of general data mining tasks, such as clustering and outlier detection in large multidimensional data sets. In density-biased sampling, the probability that a given point will be included in the sample depends on the local density of the data set. We propose a general technique for density-biased sampling that can factor in user requirements to sample for properties of interest and can be tuned for specific data mining tasks. This allows great flexibility and improved accuracy of the results over simple random sampling. We describe our approach in detail, we analytically evaluate it, and show how it can be optimized for approximate clustering and outlier detection. Finally, we present a thorough experimental evaluation of the proposed method, applying density-biased sampling on real and synthetic data sets, and employing clustering and outlier detection algorithms, thus highlighting the utility of our approach.

Outlier detection is a well-researched and crucial problem in machine learning. However, there is little research on string data outlier detection, as most literature focuses on outlier detection of numerical data. A robust string data outlier detection algorithm could assist with data cleaning or anomaly detection in system log files. In this thesis, we compare two string outlier detection algorithms. Firstly, we introduce a variant of the well-known local outlier factor algorithm, which we tailor to detect outliers on string data using the Levenshtein measure to calculate the density of the dataset. We present a differently weighted Levenshtein measure, which considers hierarchical character classes and can be used to tune the algorithm to a specific string dataset. Secondly, we introduce a new kind of outlier detection algorithm based on the hierarchical left regular expression learner, which infers a regular expression for the expected data. Using various datasets and parameters, we experimentally show that both algorithms can conceptually find outliers in string data. We show that the regular expression-based algorithm is especially good at finding outliers if the expected values have a distinct structure that is sufficiently different from the structure of the outliers. In contrast, the local outlier factor algorithms are best at finding outliers if their edit distance to the expected data is sufficiently distinct from the edit distance between the expected data.

The satellite missions CHAMP, GRACE, and GOCE mark the beginning of a new era in gravity field determination and modeling. They provide unique models of the global stationary gravity field and its variation in time. Due to inevitable measurement errors, sophisticated pre-processing steps have to be applied before further use of the satellite measurements. In the framework of the GOCE mission, this includes outlier detection, absolute calibration and validation of the SGG (satellite gravity gradiometry) measurements, and removal of temporal effects. In general, outliers are defined as observations that appear to be inconsistent with the remainder of the data set. One goal is to evaluate the effect of additive, innovative and bulk outliers on the estimates of the spherical harmonic coefficients. It can be shown that even a small number of undetected outliers (<0.2 of all data points) can have an adverse effect on the coefficient estimates. Consequently, concepts for the identification and removal of outliers have to be developed. Novel outlier detection algorithms are derived and statistical methods are presented that may be used for this purpose. The methods aim at high outlier identification rates as well as small failure rates. A combined algorithm, based on wavelets and a statistical method, shows best performance with an identification rate of about 99%. To further reduce the influence of undetected outliers, an outlier detection algorithm is implemented inside the gravity field solver (the Quick-Look Gravity Field Analysis tool was used). This results in spherical harmonic coefficient estimates that are of similar quality to those obtained without outliers in the input data.

Outlier detection techniques are well known for identifying a small amount of data objects named outliers that are far away from clusters and exist in sparse regions of data space. However, most outlier detection algorithms based on k nearest neighbors are sensitive to parameter k. The outlier detection algorithms based on clustering rely on specific clustering algorithms, and outliers are by-products. To partially circumvent these problems, motivated by the medoid concept of K-medoids clustering algorithms, in this paper, we propose a k-nearest neighbor medoid-based outlier detection method that is easy to implement and can provide competing performances with existing solutions. At the same time, a method to determine the parameter k is proposed in combination with the outlier detection algorithm proposed in this paper. Experiments performed on real datasets demonstrate the efficacy of our method.

Statistical depth functions provide from the "deepest" point a "center-outward ordering" of multidimensional data. In this sense, depth functions can measure the "extremeness" or "outlyingness" of a data point with respect to a given data set. Hence, they can detect outliers--observations that appear extreme relative to the rest of the observations. Of the various statistical depths, the spatial depth is especially appealing because of its computational efficiency and mathematical tractability. In this article, we propose a novel statistical depth, the kernelized spatial depth (KSD), which generalizes the spatial depth via positive definite kernels. By choosing a proper kernel, the KSD can capture the local structure of a data set while the spatial depth fails. We demonstrate this by the half-moon data and the ring-shaped data. Based on the KSD, we propose a novel outlier detection algorithm, by which an observation with a depth value less than a threshold is declared as an outlier. The proposed algorithm is simple in structure: the threshold is the only one parameter for a given kernel. It applies to a one-class learning setting, in which "normal" observations are given as the training data, as well as to a missing label scenario, where the training set consists of a mixture of normal observations and outliers with unknown labels. We give upper bounds on the false alarm probability of a depth-based detector. These upper bounds can be used to determine the threshold. We perform extensive experiments on synthetic data and data sets from real applications. The proposed outlier detector is compared with existing methods. The KSD outlier detector demonstrates a competitive performance.

Outlier detection in the high-dimensional data stream is a challenging data mining task. In high-dimensional data, the distance-based measures of outlierness become less effective and unreliable. Angle-based outlier detection ABOD technique was proposed as a more suitable scheme for high-dimensional data. However, ABOD is designed for static datasets and its naive application on a sliding window over data streams will result in poor performance. In this research, we propose two incremental algorithms for fast outlier detection based on an outlier threshold value in high-dimensional data streams: IncrementalVOA and \(VOA^{*}\). IncrementalVOA is a basic incremental algorithm for computing outlier factor of each data point in each window. \(VOA^{*}\) enhances the incremental computation by using a bound-based pruning method and a retrospect-based incremental computation technique. The effectiveness and efficiency of the proposed algorithms are experimentally evaluated on synthetic and real world datasets where \(VOA^{*}\) outperformed other methods.

The goal of outlier detection is to pinpoint data points that exhibit notable deviations from the rest of the observed values. It has found successful application in numerous fields, including process inspection, anti-terrorist operations, and public security. However, the majority of existing outlier algorithms rely on methods involving filling or deleting missing data, with few directly addressing incomplete data. This article studies outlier detection for incomplete real-valued data based on class-consistent technology, rough set theory, and granular computing. First, a tolerance relation founded on class-consistent technology is presented to illustrate the similarity among information values within an incomplete real-valued information system (IRVIS). Then, the tolerance classes are established based on the tolerance relation and utilized for computing approximate accuracy and other metrics. Next, an outlier factor is defined, considering both the degree of outlierness and the weight function assigned to each object within an IRVIS, elucidating the uncertainty and degree of outlier. Finally, an outlier detection algorithm (ODIRG) for an IRVIS based on class-consistent technology, rough set theory, and granular computing is devised. Numerical experiments on seven UCI datasets are undertaken to evaluate the stability of the ODIRG algorithm. The proposed method is demonstrated to exhibit strong effectiveness and adaptability for categorical data when compared with five other algorithms. It is notable that for comprehensive comparison, precision, recall, F F 1-measure, and receiver operating characteristic curve are employed to delineate the benefits of the proposed approach.

Outlier detection is one of the major data mining methods. This paper proposes a three-step approach to detect spatio-temporal outliers in large databases. These steps are clustering, checking spatial neighbors, and checking temporal neighbors. In this paper, we introduce a new outlier detection algorithm to find small groups of data objects that are exceptional when compared with rest large amount of data. In contrast to the existing outlier detection algorithms, new algorithm has the ability of discovering outliers according to the non-spatial, spatial and temporal values of the objects. In order to demonstrate the new algorithm, this paper also presents an example application using a data warehouse

Outlier Detection is a key technique to discover abnormal patterns from mass data. Pivot based metric space outlier detection algorithm is designed to improve the cutoff value of outlier degree faster and terminate the detection program earlier, for saving detection time. However, due to the lack of efficient pivot selection method, the performance of existing related algorithms is not as good as it might have been. In this paper, we propose an adaptive cutoff distance based density peak pivot selection algorithm (ADP) to get a suitable pivot quickly. Moreover, we also develop an improved outlier detection algorithm based on ADP algorithm (ADPOD). Experimental results indicate that ADPOD can effectively overcome the existing relate algorithm, and achieves a 2.67 speed up over it on average and, in certain cases, up to 8.06. The distance calculation times are reduced by 53.39% on average and up to 94.84%, as well as acceptable indexing time and the universality of metric space.

A novel battery SOH estimation algorithm based on outlier detection has been presented. The Battery state of health (SOH) is one of the most important parameters that describes the usability state of the power battery system. Firstly, a battery system model with lifetime fading characteristic was established, and the battery characteristic parameters were acquired from the lifetime fading process. Then, the outlier detection method based on angular distribution was used to identify the outliers among the battery behaviors. Lastly, the functional relationship between battery SOH and the outlier distribution was obtained by polynomial fitting method. The experimental results show that the algorithm can identify the outliers accurately, and the absolute error between the SOH estimation value and true value is less than 3%.