Outlier detection for high dimensional data

As there is an increasing demand of data, outlier detection is coming across as a popular field of research. Outlier is stated as an observation which is dissimilar from the other observations present in the data set. It is advantageous in the fields like medical industry, crime detection, fraudulent transaction, public safety etc. Outlier can be learnt in different fields like big data, time series data, high dimension data, biological data, uncertain data and many more. Most of the time 10% of the whole sample data set is incorrect, not accessible or missing sometimes. This paper studies and compares the popular outlier detection algorithms namely, Cluster based outlier detection, Distance based outlier detection and Density based outlier detection. Comparative study of these outlier detection techniques is performed to find out most efficient outlier detection method for calculation of the outlier.

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Outlier detection is critical for many applications such as healthcare, health insurance, medical diagnosis, predictive analytics, pattern recognition, intrusion detection, anomaly or defect detection, video surveillance, credit card fraud detection and text mining. Outlier detection techniques could be statistics, distance- or model based. Techniques, which are based on a single method for outlier detection usually have weaknesses and strengths and are mostly unstable. Outlier detection ensembles harness the strengths of individual detectors and result in stable performance. This paper presents a new parameter based growing self-organizing maps ensemble (GSOME) for outlier detection in multivariate patterns. For outlier detection, the proposed GSOME transforms non-linear relationships between high dimensional patterns into a simple 1D geometric relationship. Whatever the pattern dimensionality is, it is mapped to a single point of a line. The dispersion of mapped points will be used to locate the outliers and measure the degree of outlyingness. Several experiments on both real and synthetic data sets show the promising performance of the proposed GSOME.

Parallel graph-based anomaly detection technique for sequential data

In many Knowledge Discovery applications, finding outliers is more interesting than finding inliers in a dataset.The perception of outliers is rare cases in dataset in which is being described as abnormal data in the information table.Outliers detections are applied in many important applications like fraud detection systems to uncover the suspicious objects which may have important knowledge hidden in the system.A new outlier detection technique based on Rough Sets Theory (RST) is hereby proposed.RSetOF is a new measure for the outlier factor based on RST.By employing this factor, a new formulation for detecting outlier is established.The outlyingness of outliers objects in a dataset using this measurement is identified.To detect outliers, two measurements which are the top n ratio and the coverage ratio are presented.Finding top n outliers from all objects allow searching of outliers from top ranked records based on the least outlier factor value.The capability in detecting outliers at top n number of outliers will indicate how fast the detection is.The efficiency of this technique by obtaining the coverage ratio value is then tested.The maximum percentage of coverage obtained shows the maximum number of outliers detected belonging to rare cases.A comparison is hence carried out to examine the performance of the RSetAlg with a selective outlier detection method, the Frequent Pattern method referred to as FindFPOF.Ten benchmark datasets for assessing the outlier detection technique are used for this purpose.The experimental result shows that the proposed technique is competitive and proven to be better in speed of detection than the other technique.The fast and efficient detection of outliers has proven its potential as a new outliers detection technique based on RST.

The development of effective knowledge discovery techniques has become a very active research area in recent years due to the important impact it has had in several relevant application domains. One interesting task therein is that of singling out anomalous individuals from a given population, for example, to detect rare events in time-series analysis settings, or to identify objects whose behavior is deviant w.r.t. a codified standard set of rules. Such exceptional individuals are usually referred to as outliers in the literature. In this article, the concept of outlier is formally stated in the context of knowledge-based systems, by generalizing that originally proposed in Angiulli et al. [2003] in the context of default theories. The chosen formal framework here is that of logic programming, wherein potential applications of techniques for outlier detection are thoroughly discussed. The proposed formalization is a novel one and helps to shed light on the nature of outliers occurring in logic bases. Also the exploitation of minimality criteria in outlier detection is illustrated. The computational complexity of outlier detection problems arising in this novel setting is also thoroughly investigated and accounted for in the paper. Finally, rewriting algorithms are proposed that transform any outlier detection problem into an equivalent inference problem under stable model semantics, thereby making outlier computation effective and realizable on top of any stable model solver.

Outlier detection in high-dimensional space is a hot topic in data mining, the main goal is to find out a small quantity of data objects with abnormal behavior in data set. In this paper, the concepts of the feature vector and the attribute similarity are defined, an improved algorithm SWHOT based on weighed hypergraph model for outlier detection in high dimensional space is presented. The objects in high dimensional space are translated into binary data type, by looking for the hyperedge of binary set, the data set hypergarph model is established, meanwhile, the weight of the hyperedge is equal to the value of the attribute similarity. In addition, the objects of the hypergraph are clustered by CURE algorithm, arbitrary shaped clusters can be identified. Furthermore, the outliers are found according to the point-to-window weighted support, the point-to-class belongingness and the point-to-window weighted deviation of size, the meaningful outliers in high-dimension can be mined by means of appropriate user-defined threshold. Experimental results show that SWHOT can improve scaling and precision.

In this work, a comprehensive framework for traditional outlier detection techniques based on simple and multiple linear regression models was studied. Two data sets were used for the illustration and evaluation of each class of outlier detection techniques (analytical and graphical methods). Outlier detection aims at identifying such outlier in order to improve the analytic of data and suitable model built. Furthermore, comparisons of the different methods were done to highlight the advantages, disadvantages and performance issues of each class of outlier detection techniques. The results show that by removing the influential points (or outliers), the model adequacy increased (from R2 = 0.72 to R2 = 0.97). It was observed that Jackknife residuals and Atkinson’s measure methods are very useful in detecting outliers; hence, both methods were recommended for outliers’ detection.

The outlier detection technique is widely used in the data analysis for the clustering of data. Many techniques have been applied in the outlier detection to increase the efficiency of the data analysis. The Local Projection based Outlier Detection (LPOD) method effectively identifies neighbouring values of data, but this has the drawback of random selection of the cluster centre that affects the overall clustering performance of the system. In this study, the Adaptive Clustering by Fast Search and Find of Density Peak (ACFSFDP) is proposed to select the clustering centre and density peak. This ACFSFDP method is implemented with the min-max algorithm to find the number of categories that measured the local density and distance information. The density and distance are used to select the cluster centre, but density is not calculated on the existing distance based clustering techniques. The ACFSFDP method calculates cluster centre based on the density and distance during the clustering process, whereas the existing techniques randomly select the data centre. The results indicated that the ACFSFDP method is provided effective outlier detection compared with existing Clustering by Fast Search and Find of Density Peak (CFSFDP) methods. The ACFSFDP is tested on two datasets Pen-digits and waveform datasets. The experiment results proved that Area Under Curve (AUC) of the ACFSFDP is 99.08% on the Pen-Digit dataset, while the existing distance classifier method k-Nearest Neighbour has achieved 68.7% of AUC.

The rapid growth of the internet and digital communication, network security has become a critical concern. Traditional Intrusion Detection Systems (IDS), especially rule-based or signature-based systems, are effective only against known attacks [1], [2]. However, they fail to detect novel or zero-day threats because they rely on predefined signatures [5], [7]. To overcome this limitation, anomaly detection techniques have gained importance. Anomaly-based systems monitor network traffic and identify deviations from normal behavior, which could indicate potential cyber-attacks [6]. This project focuses on Network Anomaly Detection using Deep Learning techniques [3]. The proposed system leverages models such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), or Long Short-Term Memory (LSTM) networks to learn patterns of normal network traffic and accurately identify abnormal or malicious activities [4], [5]. Deep learning models are capable of handling large-scale, high-dimensional data, and can automatically extract complex features without manual intervention [3], [6]. The objective of this work is to design a robust, intelligent intrusion detection framework that can not only detect known threats but also adapt to new and evolving attack patterns. The system will be trained and validated using benchmark datasets like NSL-KDD or CICIDS2017 [1], [2]. Performance will be measured using metrics such as accuracy, precision, recall, and F1-score. By implementing deep learning-based anomaly detection, this project aims to enhance the efficiency of network security, minimize false positives, and provide a scalable solution for real-time intrusion detection in modern networks [5], [6], [7].

The comprehension of the clustering result is a problem that hasn't yet resolve, which having important meaning to the usage of the cluster result and the evaluation of the cluster effect. We put forward the method discovering attribute feature cluster for any clustering result based on outlier detection technique, and put forward an outlier detection algorithm based on even distribution pattern. Through carrying on outlier analysis to all data cluster attribute descriptions, we discovered the feature attribute of each data cluster, and then carried out the comprehension to the clustering result. The remarkable point lies in the method doesn't only aim at a particular clustering algorithm, but also the analysis of any the clustering algorithm result. Experiment to the UCI data set indicated, the method submitted in this paper obtained better result.

Destructive cyber-attacks and cybercriminals are increasing with the increase in IoT (Internet of Things) devices globally. This ha/s led to the need for increase in security in IoT systems. Innovative and novel techniques are used by the intruders to accomplish malicious goals effectively through cyber-attacks. An Intrusion Detection System (IDS) is used for classification of attacks in IoT networks based on anomaly detection and machine learning techniques. Inefficiency is observed in the conventional machine learning models and intrusion detection techniques as the network technologies are unpredictable. Accurate identification of various anomalies is possible with deep learning models in several research segments. The input data along with its prominent characteristics may be categorized automatically for classification and anomaly detection using convolutional neural networks (CNN). Faster computations are enabled due to the performance efficiency of CNN. For IoT networks, an intrusion detection model based on anomaly detection is designed and developed in this paper. A multiclass classification framework is created initially using a CNN model. Further, 3D CNN is used for implementation of the proposed model. Various intrusion detection datasets from IoT networks are used for validation of the proposed CNN model. Pre-trained multiclass CNN model is used for implementation of multiclass and binary classification based on transfer learning. When compared to the conventional deep learning models, the proposed multiclass and binary classification framework has attained improved F1 score, recall, precision and accuracy.

Outlier detection is an important data mining task for consistency checks, fraud detection, etc. Binary decision making on whether or not an object is an outlier is not appropriate in many applications and moreover hard to parametrize. Thus, recently, methods for outlier ranking have been proposed. Determining the degree of deviation, they do not require setting a decision boundary between outliers and the remaining data. High dimensional and heterogeneous (continuous and categorical attributes) data, however, pose a problem for most outlier ranking algorithms. In this work, we propose our OutRank approach for ranking outliers in heterogeneous high dimensional data. We introduce a consistent model for different attribute types. Our novel scoring functions transform the analyzed structure of the data to a meaningful ranking. Promising results in preliminary experiments show the potential for successful outlier ranking in high dimensional data.

A hybrid approach to outlier detection in the offset lithographic printing process

An outlier is an unexpected event or entity. For example, many experts view the Great Financial Crash of 2008 as an outlier event which has triggered a reappraisal of mainstream economic and financial models which define the “expected.” The objective of Outlier Detection in Data Mining is in similar vein—outliers often embody new information, which is often hard to explain in the context of existing knowledge and results in a re-evaluation of what is known. An important theme in Statistics/Machine Learning/Data Mining is the use of data to study the “norm” or “expected” behaviour of the underlying phenomenon (which generated the data). The presence of outliers often distorts the understanding of the norm and has given rise to a set of techniques, often called robust statistics, which discount the effect of outliers. A canonical example is the use of the median (which is less sensitive to outliers) as opposed to the mean (which is extremely sensitive to outliers) to characterize average behaviour. In Data Mining, an outlier is a primary object of study which can potentially lead to the discovery of new “knowledge.” Thus the emphasis has been on the design of algorithms to find outliers in complex scenarios while relaxing as many assumptions on the underlying data generating model as possible. Or more simply, the focus in Data Mining is on non-parametric (and semi-parametric) outlier detection techniques. Here is a simple example: Let D be a multi-variate data set and the objective it to discover whether there are any outliers in D. If we assume that Dwas generated from amulti-dimensional Normal distribution then it is well known that theMahalanobis distance from data points to the