DWOF: A Robust Density-Based Outlier Detection Approach

To address the problem of unsupervised outlier detection in wireless sensor networks, we develop an algorithm that (1) is flexible with respect to the outlier definition, (2) works in-network with a communication load proportional to the outcome, and (3) reveals its outcome to all sensors. We examine the algorithm’s performance using simulation with real sensor data streams. Our results demonstrate that the algorithm is accurate and imposes a reasonable communication load and level of power consumption.

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are high dimensional domains in which the data can contain hundreds of dimensions. Many recent algorithms use concepts of proximity in order to find outliers based on their relationship to the rest of the data. However, in high dimensional space, the data is sparse and the notion of proximity fails to retain its meaningfulness. In fact, the sparsity of high dimensional data implies that every point is an almost equally good outlier from the perspective of proximity-based definitions. Consequently, for high dimensional data, the notion of finding meaningful outliers becomes substantially more complex and non-obvious. In this paper, we discuss new techniques for outlier detection which find the outliers by studying the behavior of projections from the data set.

Outlier detection is the task of automatic identification of unknown data not covered by training data (e.g. a previously unknown class in classification). We explore outlier detection in the presence of hubs and anti-hubs, i.e. data objects which appear to be either very close or very far from most other data due to a problem of measuring distances in high dimensions. We compare a classic distance based method to two new approaches, which have been designed to counter the negative effects of hubness, on six high-dimensional data sets. We show that mainly anti-hubs pose a problem for outlier detection and that this can be improved by using a hubness-aware approach based on re-scaling the distance space.

In many Knowledge Discovery applications, finding outliers is more interesting than finding inliers in a dataset.The perception of outliers is rare cases in dataset in which is being described as abnormal data in the information table.Outliers detections are applied in many important applications like fraud detection systems to uncover the suspicious objects which may have important knowledge hidden in the system.A new outlier detection technique based on Rough Sets Theory (RST) is hereby proposed.RSetOF is a new measure for the outlier factor based on RST.By employing this factor, a new formulation for detecting outlier is established.The outlyingness of outliers objects in a dataset using this measurement is identified.To detect outliers, two measurements which are the top n ratio and the coverage ratio are presented.Finding top n outliers from all objects allow searching of outliers from top ranked records based on the least outlier factor value.The capability in detecting outliers at top n number of outliers will indicate how fast the detection is.The efficiency of this technique by obtaining the coverage ratio value is then tested.The maximum percentage of coverage obtained shows the maximum number of outliers detected belonging to rare cases.A comparison is hence carried out to examine the performance of the RSetAlg with a selective outlier detection method, the Frequent Pattern method referred to as FindFPOF.Ten benchmark datasets for assessing the outlier detection technique are used for this purpose.The experimental result shows that the proposed technique is competitive and proven to be better in speed of detection than the other technique.The fast and efficient detection of outliers has proven its potential as a new outliers detection technique based on RST.

Unsupervised outlier detection using random subspace and subsampling ensembles of Dirichlet process mixtures

The development of effective knowledge discovery techniques has become a very active research area in recent years due to the important impact it has had in several relevant application domains. One interesting task therein is that of singling out anomalous individuals from a given population, for example, to detect rare events in time-series analysis settings, or to identify objects whose behavior is deviant w.r.t. a codified standard set of rules. Such exceptional individuals are usually referred to as outliers in the literature. In this article, the concept of outlier is formally stated in the context of knowledge-based systems, by generalizing that originally proposed in Angiulli et al. [2003] in the context of default theories. The chosen formal framework here is that of logic programming, wherein potential applications of techniques for outlier detection are thoroughly discussed. The proposed formalization is a novel one and helps to shed light on the nature of outliers occurring in logic bases. Also the exploitation of minimality criteria in outlier detection is illustrated. The computational complexity of outlier detection problems arising in this novel setting is also thoroughly investigated and accounted for in the paper. Finally, rewriting algorithms are proposed that transform any outlier detection problem into an equivalent inference problem under stable model semantics, thereby making outlier computation effective and realizable on top of any stable model solver.

Three-way decision-based co-detection for outliers

The paper presents the problem of outlier detection in the rule knowledge bases. Unusual (rare) rules, regarded here as the deviation, should be the subject of analysis experts and knowledge engineers because they can influence the efficiency of inference in decision support systems. The author presents a different approach in finding outliers in the rules. The experiments with their results are also presented in the paper.

Outlier detection is a very important task in many fields like network intrusion detection, credit card fraud detection, stock market analysis, detecting outlying cases in medical data etc. Outlier detection in streaming data is very challenging because streaming data cannot be scanned multiple times and also new concepts may keep evolving in coming data over time. Irrelevant attributes can be termed as noisy attributes and such attributes further magnify the challenge of working with data streams. In this paper, we propose an unsupervised outlier detection scheme for streaming data. This scheme is based on clustering as clustering is an unsupervised data mining task and it does not require labeled data. In proposed scheme both density based and partitioning clustering method are combined to take advantage of both density based and distance based outlier detection. Proposed scheme also assigns weights to attributes depending upon their respective relevance in mining task and weights are adaptive in nature. Weighted attributes are helpful to reduce or remove the effect of noisy attributes. Keeping in view the challenges of streaming data, the proposed scheme is incremental and adaptive to concept evolution. Experimental results on synthetic and real world data sets show that our proposed approach outperforms other existing approach (CORM) in terms of outlier detection rate, false alarm rate, and increasing percentages of outliers.

With the widespread use of closed-circuit television (CCTV) surveillance systems in public areas, crowd anomaly detection has become an increasingly critical aspect of the intelligent video surveillance system. It requires workforce and continuous attention to decide on the captured event, which is hard to perform by individuals. The available literature on human action detection includes various approaches to detect abnormal crowd behavior, which is articulated as an outlier detection problem. This paper presents a detailed review of the recent development of anomaly detection methods from the perspectives of computer vision on different available datasets. A new taxonomic organization of existing works in crowd analysis and anomaly detection has been introduced. A summarization of existing reviews and datasets related to anomaly detection has been listed. It covers an overview of different crowd concepts, including mass gathering events analysis and challenges, types of anomalies, and surveillance systems. Additionally, research trends and future work prospects have been analyzed.

Outlier detection is to separate anomalous data from inliers in the dataset. Recently, the most deep learning methods of outlier detection leverage an auxiliary reconstruction task by assuming that outliers are more difficult to recover than normal samples (inliers). However, it is not always true in deep auto-encoder (AE) based models. The auto-encoder based detectors may recover certain outliers even if outliers are not in the training data, because they do not constrain the feature learning. Instead, we think outlier detection can be done in the feature space by measuring the distance between outliers' features and the consistency feature of inliers. To achieve this, we propose an unsupervised outlier detection method using a memory module and a contrastive learning module (MCOD). The memory module constrains the consistency of features, which merely represent the normal data. The contrastive learning module learns more discriminative features, which boosts the distinction between outliers and inliers. Extensive experiments on four benchmark datasets show that our proposed MCOD performs well and outperforms eleven state-of-the-art methods.

Restricted default theories: Expressive power and outlier detection tasks

Identifying unusual or anomalous patterns in an underlying dataset is an important but challenging task in many applications. The focus of the unsupervised anomaly detection literature has mostly been on vectorised data. However, many applications are more naturally described using higher-order tensor representations. Approaches that vectorise tensorial data can destroy the structural information encoded in the high-dimensional space, and lead to the problem of the curse of dimensionality. In this paper we present the first unsupervised tensorial anomaly detection method, along with a randomised version of our method. Our anomaly detection method, the One-class Support Tensor Machine (1STM), is a generalisation of conventional one-class Support Vector Machines to higher-order spaces. 1STM preserves the multiway structure of tensor data, while achieving significant improvement in accuracy and efficiency over conventional vectorised methods. We then leverage the theory of nonlinear random projections to propose the Randomised 1STM (R1STM). Our empirical analysis on several real and synthetic datasets shows that our R1STM algorithm delivers comparable or better accuracy to a state-of-the-art deep learning method and traditional kernelised approaches for anomaly detection, while being approximately 100 times faster in training and testing.

It is the main task of visual anomaly detection to find local regions whose saliency is inconsistent with normal appearance. However, existing mainstream anomaly detection models suffer from low detection accuracy and poor generalization performance. Therefore, this paper designs an unsupervised surface anomaly detection model based on attention and atrous spatial pyramid pooling. The proposed model learns anomaly images and their normal reconstruction and simultaneously learns the decision boundaries of normal and anomaly images. The method utilizes a squeeze-and-excitation block to assign attention to feature channels to improve the sensitivity of related favorable features, thus enhancing the model’s ability to learn normal and anomaly boundaries. In addition, atrous spatial pyramid pooling is introduced in the discriminative sub-network to obtain the multi-scale semantic information of the training image, which improves the detection ability of defects of different sizes and enhances the universality of the model. The superiority of our method is demonstrated in the anomaly detection benchmark MVTec dataset.

In this paper, an outlier detection method based on kernel-density-fluctuation is proposed, which is suitable for preprocessing transformer load monitoring data and can improve the quality of data sets used for data mining tasks. At first, some basic concepts involved in this newly proposed algorithm are introduced, and then the definition and calculation method of the k-density-fluctuation outlier factor are illustrated, which is used to quantify the possibility that this object is an anomaly. Finally, the algorithm rearranges the objects in the origin data set according to the value of their k-density-fluctuation outlier factors to discover the outliers. The simulation results of the algorithm running on the artificial data set and the natural data set to verify that the performance of the proposed algorithm in detecting isolated points is more ideal than the current used algorithms such as LOF in preprocessing transformer load monitoring data.