Outlier detection for mixed data

Outliers detection has wide application for financial surveillance. The Traditional outlier detection method is based on statistical models, such as ARMA, ARCH and GARCH, which require special hypotheses, and they are inappropriate to apply to complex financial data, such as high frequency data. This paper introduces a new data mining method to detect outliers for analysis of share index fluctuation. Based on the Voronoi diagram, we propose a novel outlier detection method, which called Voronoi based Outlier Detection (VOD). Experiments show the VOD method performs more efficient and effective against the existing method in outlier detection for financial data.

Outlier detection is one of the most important data mining techniques. It has broad applications like fraud detection, credit approval, computer network intrusion detection, anti-money laundering, etc. The basis of outlier detection is to identify data points which are “different” or “far away” from the rest of the data points in the given dataset. Traditional outlier detection method is based on statistical analysis. However, this traditional method has an inherent drawback—it requires the availability of the entire dataset. In practice, especially in the real time data feed application, it is not so realistic to wait for all the data because fresh data are streaming in very quickly. Outlier detection is hence done in batches. However two drawbacks may arise: relatively long processing time because of the massive size, and the result may be outdated soon between successive updates. In this paper, we propose several novel incremental methods to process the real time data effectively for outlier detection. For the experiment, we test three types of mechanisms for analyzing the dataset, namely Global Analysis, Cumulative Analysis and Lightweight Analysis with Sliding Window. The experiment dataset is “household power consumption” which is a popular benchmarking data for Massive Online Analysis.

In this study, we delve into the prediction and recommendation algorithm for user repurchase behavior within the community e-commerce landscape, leveraging the framework of information entropy. By analyzing consumer behavior data from a product-centered perspective, we observe that the majority of consumers engage in approximately 0-8 information search clicks before making a purchase, primarily focusing on product details, with an average of about 4 browsing interactions. Subsequently, a minimal portion of consumers exhibit 0-5 instances of preference behavior after browsing activities. Similarly, consumers who add items to their shopping carts typically engage in this behavior from 0 to 4 times, showcasing limited participation. Leveraging attribute probability, information entropy serves as a driving force for predicting community e-commerce users' repurchase behavior, contrasting with traditional outlier detection methods and underscoring the advantages of information entropy.

In this paper we present a "model free' method of outlier detection for Gaussian time series by using the autocorrelation structure of the time series. We also present a graphic diagnostic method in order to distinguish an additive outlier (AO) from an innovation outlier (IO). The test statistic for detecting the outlier has a P ² distribution with one degree of freedom. We show that this method works well when the time series contain either one type of the outliers or both additive and innovation type outliers, and this method has the advantage that no time series model needs to be estimated from the data. Simulation evidence shows that different types of outliers can be graphically distinguished by using the techniques proposed.

Outlier detection (or anomaly detection) is a fundamental task in data mining. Outliers are data that deviate from the norm and outlier detection is often compared to “finding a needle in a haystack.” However, the outliers may generate high value if they are found, value in terms of cost savings, improved efficiency, compute time savings, fraud reduction and failure prevention. Detection can identify faults before they escalate with potentially catastrophic consequences. Big Data refers to large, dynamic collections of data. These vast and complex data appear problematic for traditional outlier detection methods to process but, Big Data provides considerable opportunity to uncover new outliers and data relationships. This chapter highlights some of the research issues for outlier detection in Big Data and covers the solutions used and research directions taken along with an analysis of some current outlier detection approaches for Big Data applications.

Evolving anomaly detection for network streaming data

Nowadays many applications generate mixed data objects consisting of numerical and categorical attributes. Simultaneously dealing with mixed objects is more challenging and various approaches convert one type to another one to face this issue. But in many cases this leads to information loss. Therefore integrating categorical and numerical attributes sounds reasonable since it keeps the original format of any attribute. In this paper we focus on clustering and especially density-based clustering as one of the well-known clustering approaches well-performed on arbitrary shape clusters. Density-based clustering algorithms require a distance measure to discover dense regions. Therefore we introduce the distance hierarchy as a distance measure appropriate for both categorical and numerical attributes. However setting the parameters regarding any parametric clustering algorithm could be another issue. Therefore we employ minimum description length principle to automate this process.

Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis

Ground-based solar resource measurements are known to be preferred to synthetic or simulated data for a given location, but outliers present in this data can significantly impact the accuracy of predictions used in viability assessments. For solar energy installations to be self-sustaining and viable, accurate ground-based solar resource data for the location of these installations are essential for decision-making and planning. Conventional outlier detection techniques used for solar resources, including graphical plots to complex numerical approaches, often have difficulty identifying these outliers to a satisfactory degree. This study proposes the use of simulated outliers added to synthetic data to train and compare the effectiveness of traditional outlier detection methods and several statistical learning methods, including NN, naïve Bayes, support vector machines and advanced tree-based models for the purpose of outlier detection in this field. The results indicate that the advanced tree-based models provide accurate identification of outliers in the simulation step and are demonstrated to be effective on a ground-based real world data set collected in Gqeberha, South Africa. The use of the proposed approach can aid in reducing the uncertainty in measured solar resource data and, as a result, help to promote the use of solar energy solutions in areas with unreliable solar resource data.

Various data-based soft sensor models have been established for online prediction of the silicon content in a pig iron. Actually, modeling data often contain many outliers and this can deteriorate the quality of models. However, little attention is paid to efficient outlier detection. Besides, most of traditional outlier detection methods are assumed that data are distributed (approximately) normally and thus they might be invalid for some situations. A new multivariate preprocessing method for outlier detection without any assumption of data distribution is proposed. This novel outlier detection method mainly utilizes a support vector clustering (SVC) strategy. After SVC-based preprocessing, a support vector regression soft sensor model is built. A comparative study for an industrial blast furnace is investigated and the results show its superiority.

A wide range of methods have been proposed for detecting different types of outliers in both the full attribute space and its subspaces. However, the interpretability of outliers, that is, explaining in what ways and to what extent an object is an outlier, remains a critical issue. In this paper, we focus on improving the interpretability of outliers. Particularly, we develop a notion of multidimensional contextual outliers to model the context of an outlier, and propose a framework for contextual outlier detection. Intuitively, a contextual outlier is a small group of objects that share strong similarity with a significantly larger reference group of objects on some attributes, but deviate dramatically on some other attributes. In contextual outlier detection, we identify not only the outliers, but also their associated contextual information including (1) comparing to what reference group of objects the detected object(s) is/are an outlier; (2) the attributes defining the unusual behavior of the outlier(s) compared against the reference group; (3) the population of similar outliers sharing the same context; and (4) the outlier degree, which measures the population ratio between the reference group and the outlier group. We present an algorithm and conduct extensive experiments to evaluate our approach.

Clustering algorithm for mixed datasets using density peaks and Self-Organizing Generative Adversarial Networks

Various methods have been developed to detect outliers which are significantly different from others. Most outlier detection methods assume the data lie in Euclidean space in which distances can be easily defined and computed. In reality, we meet many data with both numerical and categorical attributes together, so-called mixed-data, for which it is not easy to define widely-accepted distance metrics. This paper proposes an outlier detection method which can be applied to mixed data. The method focuses on the association among attribute values. It first selects the sets of potentially associated attributes, computes the degrees of outlierness for records with respect to the associated attributes, and then determines a collection of outliers using the degrees. In addition, this paper shows some experiment results of the proposed method and compares with some other methods.

Since the Local Outlier Factor (LOF) was first proposed, there is a large family of approaches that is derived from it. For the reason that the existing local outliers detection approaches only focus on the extent of overall separation between an object and its neighbors, and do not pay attention to the degree of dispersion between them, the precision of these approaches will be seriously affected in the scattered data sets for outlier detection. In this paper, we redefine the local outliers by combining the degree of dispersion of the object and its neighbors, and propose a new local outlier detection approach (N2DLOF). Compared to conventional approaches, the outliers obtained by N2DLOF are more sensitive to the degree of anomaly of the scattered data sets. Experiments show that our approach has a significant improvement on outlier detection precision in the case of scattered datasets with similar time complexity. In short, we extend the ecosystem of the local outlier detection approaches from a new perspective.

Outlier detection is an important task in data mining with numerous applications, including credit card fraud detection, video surveillance, etc. A recent work on outlier detection has introduced a novel notion of local outlier in which the degree to which an object is outlying is dependent on the density of its local neighborhood, and each object can be assigned a Local Outlier Factor (LOF) which represents the likelihood of that object being an outlier. Although the concept of local outliers is a useful one, the computation of LOF values for every data objects requires a large number of κ-nearest neighbors searches and can be computationally expensive. Since most objects are usually not outliers, it is useful to provide users with the option of finding only n most outstanding local outliers, i.e., the top-n data objects which are most likely to be local outliers according to their LOFs. However, if the pruning is not done carefully, finding top-n outliers could result in the same amount of computation as finding LOF for all objects. In this paper, we propose a novel method to efficiently find the top-n local outliers in large databases. The concept of "micro-cluster" is introduced to compress the data. An efficient micro-cluster-based local outlier mining algorithm is designed based on this concept. As our algorithm can be adversely affected by the overlapping in the micro-clusters, we proposed a meaningful cut-plane solution for overlapping data. The formal analysis and experiments show that this method can achieve good performance in finding the most outstanding local outliers.