A Spectral Clustering Based Outlier Detection Technique

As there is an increasing demand of data, outlier detection is coming across as a popular field of research. Outlier is stated as an observation which is dissimilar from the other observations present in the data set. It is advantageous in the fields like medical industry, crime detection, fraudulent transaction, public safety etc. Outlier can be learnt in different fields like big data, time series data, high dimension data, biological data, uncertain data and many more. Most of the time 10% of the whole sample data set is incorrect, not accessible or missing sometimes. This paper studies and compares the popular outlier detection algorithms namely, Cluster based outlier detection, Distance based outlier detection and Density based outlier detection. Comparative study of these outlier detection techniques is performed to find out most efficient outlier detection method for calculation of the outlier.

Outlier detection has numerous useful applications such as detecting criminal activities in electronic commerce, terrorism prediction and exceptional cases in many areas. Privacy and security concerns, however, arise while performing mining for outliers on distributed data. In this paper, we present two privacy preserving distance-based outlier detection algorithms over vertically partitioned data, not disclosing any private information to any participant. The first is between two parties and the second among multi-parties. The security and performances such as computation and communication complexities are analyzed for both of the two privacy preserving algorithms.

Recently, many Intrusion Detection Systems (IDS) have been proposed to detect and report about malicious nodes in Mobile Ad Hoc Networks (MANETs) by using various outlier detection algorithms. All these algorithms have limitations in terms of accuracy and speed. In this paper, we propose a new outlier detection algorithm called Weighted Distance Based Outlier Detection (WDBOD) algorithm to detect the intruders in wireless environment. The experimental results show that the proposed algorithm improves the detection accuracy and reduces the false alarm rate.

Due to rapid evolution in technologies, data acquisition techniques generate a huge amount of data in real-world scenario and capture information about marketing, business, banking, government, science and technology, etc. Regardless of huge amount of data, some data point shows exceptional or abnormal behavior in a dataset which are called as outliers. Outlier detection plays a significant role in several fields like: fraud detection, medical diagnosis, intrusion detection, financial application. To detect outliers in a given dataset, firstly outlier detection algorithms build a normal pattern in a dataset and then allocate an outlying degree to each point of a dataset based on deviation from normal pattern. This paper presents a distance-based outlier detection algorithm using particle swarm optimization (PSO) technique. In this algorithm, it assigns an outlying degree to each data point using the sum of distances between points and its k-nearest neighbor set, and PSO is used to detect outlying subspaces where outliers may exist. Finally, experimental studies show that our proposed algorithm is more effective in terms of efficiency and accuracy by using UCI dataset.

In data mining, outlier detection is a major challenge as it has an important role in many applications such as medical data, image processing, fraud detection, intrusion detection, and so forth. An extensive variety of clustering based approaches have been developed to detect outliers. However they are by nature time consuming which restrict their utilization with real-time applications. Furthermore, outlier detection requests are handled one at a time, which means that each request is initiated individually with a particular set of parameters. In this paper, the first clustering based outlier detection framework, (On the Fly Clustering Based Outlier Detection (OFCOD)) is presented. OFCOD enables analysts to effectively find out outliers on time with request even within huge datasets. The proposed framework has been tested and evaluated using two real world datasets with different features and applications; one with 699 records, and another with five millions records. The experimental results show that the performance of the proposed framework outperforms other existing approaches while considering several evaluation metrics.

Outlier detection is of great significance in the domain of data mining. Its task is to find those target points that are not identical to most of the object generation mechanisms. The existing algorithms are mainly divided into density-based algorithms and distance-based algorithms. However, both approaches have some drawbacks. The former struggles to handle low-density modes, while the latter cannot detect local outliers. Moreover, the outlier detection algorithm is very sensitive to parameter settings. This paper proposes a new two-parameter outlier detection (TPOD) algorithm. The method proposed in this paper does not need to manually define the number of neighbors, and the introduction of relative distance can also solve the problem of low density and further accurately detect outliers. This is a combinatorial optimization problem. Firstly, the number of natural neighbors is iteratively calculated, and then the local density of the target object is calculated by adaptive kernel density estimation. Secondly, the relative distance of the target points is computed through natural neighbors. Finally, these two parameters are combined to obtain the outlier factor. This eliminates the influence of parameters that require users to determine the number of outliers themselves, namely, the top-n effect. Two synthetic datasets and 17 real datasets were used to test the effectiveness of this method; a comparison with another five algorithms is also provided. The AUC value and F1 score on multiple datasets are higher than other algorithms, indicating that outliers can be found accurately, which proves that the algorithm is effective.

Nowadays, GPS-enabled devices play a vital role in location-based services and daily basis activities such as: locating where your vehicle is at any given time, recovering a stolen vehicle, monitoring your children's location, and traffic or weather alerts. Indeed, precise locations generated from these GPS devices are a must to ensure that the location-based services work accurately. For this purpose, the discovery of abnormal patterns in spatio-temporal data is a significant topic for many kinds of researchers. In this paper, the methods of spatio-temporal outlier detection are categorized into four types: distance and density-based outlier detection, pattern outlier detection, supervised and semi-supervised learning, and statistical and probabilistic techniques. This chapter describes the datasets used in the approaches of spatio-temporal outlier detection and explores its popular applications. The main contribution of this study is to guide researchers to define research gaps in trajectories outlier detection and choose the proper techniques that cope with their research problems.

Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis

Outliers are the data objects or data elements which are deviated from the observation or population in the dataset. Normally outliers are considered as noise data but in recent days outliers are taken vital role in the applications like healthcare systems to energetically check for any malignant, irregular, or abnormal behaviour. Accuracy of the outlier detection is purely based upon the efficiency of outlier detection methods and application where the outlier detection is involved. In this paper we design hybrid three phase modified fuzzy c-means and diverse distance based outlier detection method for distributed dataset to detect the unusual usage of data and illegitimate approach in large-scale integral networks, especially in healthcare sectors (EPR systems). The proposed algorithm combining the features of modified C-Means Fuzzification, Z-score and Manhattan distance in outlier identification. The proposed algorithm provides efficiency in outlier detection on univariate EPRdata. This paper also mainly focuses the application intrusion detection in healthcare data. The algorithm is tested on real world dataset of machine learning database repository (UCIML).

The process of discovering interesting patterns in large, possibly huge, data sets is referred to as data mining, and can be performed in several flavours, known as "data mining functions." Among these functions, outlier detection discovers observations which deviate substantially from the rest of the data, and has many important practical applications. Outlier detection in very large data sets is however computationally very demanding and currently requires high-performance computing facilities. We propose a family of parallel and distributed algorithms for graphic processing units (GPU) derived from two distance-based outlier detection algorithms: BruteForce and SolvingSet. The algorithms differ in the way they exploit the architecture and memory hierarchy of the GPU and guarantee significant improvements with respect to the CPU versions, both in terms of scalability and exploitation of parallelism. We provide a detailed discussion of their computational properties and measure performances with an extensive experimentation, comparing the several implementations and showing significant speedups. © 2016 IEEE.

Outlier detection techniques are well known for identifying a small amount of data objects named outliers that are far away from clusters and exist in sparse regions of data space. However, most outlier detection algorithms based on k nearest neighbors are sensitive to parameter k. The outlier detection algorithms based on clustering rely on specific clustering algorithms, and outliers are by-products. To partially circumvent these problems, motivated by the medoid concept of K-medoids clustering algorithms, in this paper, we propose a k-nearest neighbor medoid-based outlier detection method that is easy to implement and can provide competing performances with existing solutions. At the same time, a method to determine the parameter k is proposed in combination with the outlier detection algorithm proposed in this paper. Experiments performed on real datasets demonstrate the efficacy of our method.

Background/Objectives: Different methods can be applied to create predictive models for the clinical data with binary outcome variable. This research aims to explore the process of constructing the modified predictive model of Logistic Regression (LR). Method/Statistical Analysis: To improve the accuracy of prediction, the Distance based Outlier Detection (DBOD)is used for pre-processing and Bipolar Sigmoid Function calculated using Neuro based Weight Activation Function is used in Logistic Regression instead of Sigmoid Function. Datasets were collected from clinical laboratory of AR Hospital in Madurai for the three years 2012, 2013 and 2014 are used for analysis. Data pre-processing is done to avoid the existence of insignificant data in the dataset. The detected outliers, using DBOD method are treated using a method closest to the normal range. A comparative study among different distance measures likes Euclidean and Manhattan etc. are done for DBOD method. The pre-processed data finally is fed as input to the Logistic Regression model. Maximum likelihood estimation is used to fit the model. Logistic Model is built from the Sigmoid Function using the Regression Coefficients. The accuracy of the model is evaluated by 10 fold cross validation. Findings: Logistic Model is built from the Sigmoid Function using the Regression Coefficients, produces the accuracy of 79%. The Sigmoid Function calculated using Random Weight Function provides the prediction accuracy of 84.2% and the Bipolar Sigmoid Function calculated using Neuro based Weight Activation function provides the prediction accuracy of 90.4%. On comparison, Bipolar Sigmoid Function calculated using Neuro weight activation function outperforms well than the Sigmoid Function calculated using regression coefficients. Improvements/Applications: The accuracy of Logistic Regression is improved from 79% to 90.4%. The most important factors: Erythrocyte Sedimentation Rate (ESR) and Estimation of Mean blood Glucose are identified from positive subjects of Diabetes Mellitus. The analysis is done for the 31 Diabetes Disease attributes of three years dataset. Keywords: Bipolar Sigmoid Neuro-Weight Activation Function, Distance based Outlier Detection Method, Logistic Regression, Random Weight Function, Sigmoid Activation Function, Type 2 Diabetes Risk Factors

Many real-world use cases benefit from fast training and prediction times, and much research went into speeding up distance-based outlier detection methods to millions of data points. Contrary to popular belief, our findings suggest that little data is often enough for distance-based outlier detection models. We show that using only a tiny fraction of the data to train distance-based outlier detection models often leads to no significant reduction in predictive performance and detection variance over a wide range of tabular datasets. Furthermore, we compare a data reduction based on random subsampling and clustering-based prototypes and show that both approaches yield similar outlier detection results. Simple random subsampling, thus, proves to be a useful benchmark and baseline for future research on speeding up distance-based outlier detection.

Recently, outlier detection has widespread applications in different areas. The task is to identify outliers in the dataset and extract potential information. The existing outlier detection algorithms mainly do not solve the problems of parameter selection and high computational cost, which leaves enough room for further improvements. To solve the above problems, our paper proposes a parameter-free outlier detection algorithm based on dataset optimization method. Firstly, we propose a dataset optimization method (DOM), which initializes the original dataset in which density is greater than a specific threshold. In this method, we propose the concepts of partition function (P) and threshold function (T). Secondly, we establish a parameter-free outlier detection method. Similarly, we propose the concept of the number of residual neighbors, as the number of residual neighbors and the size of data clusters are used as the basis of outlier detection to obtain a more accurate outlier set. Finally, extensive experiments are carried out on a variety of datasets and experimental results show that our method performs well in terms of the efficiency of outlier detection and time complexity.

In order to solve the problem that the existing outlier detection algorithm is difficult to detect the one-dimensional integer data set with uneven frequency distribution and uniform distance distribution and low accuracy, the advantages of density outlier detection and distance outlier detection can be combined. An outlier detection algorithm DAD (Density and Distance) based on density and distance was proposed. In order to improve the possibility of outlier sample distance, the algorithm can define the weight distance; introduce the global density outlier factor combined with the weight distance as the relative distance, and use the cutting edge strategy to quickly cut the outliers based on the minimum spanning tree. Then, an artificial data set was used to test the algorithm. Experimental results showed that the algorithm had a good outlier detection accuracy in dealing with data sets with uneven frequency distribution and uniform distance distribution.