Anomaly Detection Using Proximity Graph and PageRank Algorithm

The objective of the study summarized in this paper was to establish a robust method to detect outliers in rate and pressure data used for decline analysis and forecasting, pressure and rate transient analysis, and similar workflows. Presence of outliers contributes significantly to the uncertainty and non-uniqueness associated with the rate/pressure transient analysis workflow. In time-series data usually encountered in rate/pressure transient analysis workflows, we observed that the neighboring data points of an inlying data point will be closer to the point as compared to closeness of an outlier data point. The outlier detection method we developed exploits this observation. It relies on the concept of local density around a data point, and locality is quantified by a function of distance of the data point from its k-nearest neighbors. This concept further maps into a local outlier factor, which signifies the quantitative extent of a data point being an outlier. Outliers will have a higher value of local outlier factor than will inlying data points. A threshold value of local outlier factor is used to label the data points as outliers, and this threshold is set using the histogram of local outlier factor values. Most commonly used outlier detection methods in the industry assume that rate/pressure data follow a known model (e.g., a Gaussian probability distribution model), but outliers deviate strongly from this model. In the usual parametric-model-based approach, an a priori assumption of the correct model needs to be verified, which can limit the validity of the approach. The newly proposed outlier detection method, being non-parametric, is more robust as it is independent of the assumption that data follows a known model. We present a synthetic case study demonstrating the importance of outlier detection and outlining the limitations of currently used outlier detection methods. We then validate our method through synthetic examples generated using numerical models of multi-stage hydraulically fractured wells in unconventional reservoirs. Upon validation we demonstrate application of our method using field examples from four major shale plays.

Outlier detection is a nontrivial but important task for many of the application areas. There exist several methods in literature to find outliers. However, there is no single method that outperforms in all cases. Thus, finding proper algorithm and the value of its relevant parameter is crucial. In addition, none of the methods are perfect and verification by the domain experts can confirm if the outliers detected are meaningful or not. Proper visual representation of the detected outliers may help experts to resolve anomalies. In this paper, we proposed a visual analytic system that finds proper algorithm and value of its relevant parameter for a specific dataset using training set. Later, the chosen method is applied to the test data to obtain the outlier ranking. After that, data points are visualized with parallel coordinate plot (PCP) where colors of the lines are obtained by using the outlier factor of the data points. PCP is one of the popular high-dimensional data visualization techniques where coordinates are parallel to each other and each data point is represented by a line. Using the visual, experts can provide feedback and update the result. Experiments with different datasets ensure the strength of our system.

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

This paper provides a new similarity detection algorithm. Given an input set of multi-dimensional data points, where each data point is assumed to be multi-dimensional, and an additional reference data point for similarity finding, the algorithm uses kernel method that embeds the data points into a low dimensional manifold. Unlike other kernel methods, which consider the entire data for the embedding, our method selects a specific set of kernel eigenvectors. The eigenvectors are chosen to separate between the data points and the reference data point so that similar data points can be easily identified as being distinct from most of the members in the dataset.

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

Surface display techniques, including contour mapping, usually use an interpolant when data values are on a rectangular grid; but when values are irregularly located some form of assumed or estimated distance-weighting function is often used to estimate the influence of data points at other locations. If moving average (or some other) techniques are then applied, slopes may be zero at data points; extrema may fall at data points exclusively; or surfaces may not pass through data points.An alternative method is proposed whereby a map is automatically divided into suitable triangular domains with a data point at each vertex. Surface estimation and contour plotting are then performed independently for each triangle using a measured or best-fit plane associated with each data point. The approach requires the utilization of three non-original aspects: 1) the derivation of a local homogeneous area coordinate system for any arbitrary triangle; 2) the construction of a data-structure linking each triangular domain with its three neighbouring triangles and three associated data points, and 3) the use of a conforming triangular finite-element interpolating function.Use of the first two of these concepts permits the economic generation and optimization of a triangular mesh from a set of data points. Optimization criteria used to define the best triangular partition are described in some detail, along with computer timing for this step. Use of the first and third concepts permits the interpolation of a smooth surface over the whole map even though each triangular element is estimated and plotted independently. The requirements for a suitable interpolating function are discussed, and an interpolant is suggested that preserves elevation and slope at each data point as well as elevation and slope continuity between domains. Extrema need not be located at data points but are constrained by their associated planes. Contour line segments are produced by division of each triangle into N2 sub-triangles, elevation estimation at each resulting node, and linear interpolation and plotting withing each sub-triangle. Resolution is then a function of N and local data point density. Extension of the method to higher dimensions is briefly discussed.

The functions of multiple instances (FUMI) approach for learning target and nontarget signatures is introduced. FUMI is a generalization of the multiple-instance learning (MIL) approach for supervised learning. FUMI differs significantly from standard MIL and supervised learning approaches because only data points which are functions of class concepts/signatures are available. In particular, this paper addresses the problem in which data points are convex combinations of target and nontarget signatures. Two algorithms, convex FUMI ( $c$ FUMI) and extended $c$ FUMI ( $e$ FUMI) , are presented and applied to the problem of hyperspectral unmixing and target detection. $c$ FUMI learns target and nontarget signatures (i.e., target and nontarget endmembers) , the number of nontarget signatures, and the proportion of each signature for every data point. The $e$ FUMI algorithm extends the $c$ FUMI to allow for additional “bag level” uncertainty in training labels. For these methods, training data need only binary labels indicating whether a data point (or some spatial area in the case of $e$ FUMI) contains or does not contain some proportion of the target; the specific target proportions for the training data are not needed. After learning the target signature using the binary-labeled training data, target detection can be performed on test data. Results for subpixel target detection on simulated and real airborne hyperspectral data are shown.

Similarity search plays an important role in many applications involving high-dimensional data. Due to the known dimensionality curse, the performance of most existing indexing structures degrades quickly as the feature dimensionality increases. Hashing methods, such as locality sensitive hashing (LSH) and its variants, have been widely used to achieve fast approximate similarity search by trading search quality for efficiency. However, most existing hashing methods make use of randomized algorithms to generate hash codes without considering the specific structural information in the data. In this paper, we propose a novel hashing method, namely, robust hashing with local models (RHLM), which learns a set of robust hash functions to map the high-dimensional data points into binary hash codes by effectively utilizing local structural information. In RHLM, for each individual data point in the training dataset, a local hashing model is learned and used to predict the hash codes of its neighboring data points. The local models from all the data points are globally aligned so that an optimal hash code can be assigned to each data point. After obtaining the hash codes of all the training data points, we design a robust method by employing l2,1 -norm minimization on the loss function to learn effective hash functions, which are then used to map each database point into its hash code. Given a query data point, the search process first maps it into the query hash code by the hash functions and then explores the buckets, which have similar hash codes to the query hash code. Extensive experimental results conducted on real-life datasets show that the proposed RHLM outperforms the state-of-the-art methods in terms of search quality and efficiency.

All data presented pertain to 11 advertised brands of same low-priced, high-consumption food product. The data span an eight-year period with nine data points for each brand. Two measurements were taken during first year with about a four-month interval between readings; remaining seven measurements were obtained at approximate yearly intervals. The 11 brands came from three separate markets, and each reported value was based on no fewer than 200 principal female shoppers in a family who used product category. Sampling was of two types: (1) replicated probability samples of blocks with clusters split proportionally between central city and surrounding suburban towns or (2) random samples from metropolitan telephone directory with specification of individual telephone numbers. The share-of-families data were obtained from question: of do you buy? If respondents had difficulty with usually buy, interviewers were instructed to add: the you buy most often. Share of families, for study's food category, is judged to be a reasonable estimate of market share based on an additional analysis. The surveys also included a question on units purchased per week. These data, in recent surveys, have been multiplied by users to develop individual volume shares. The resulting percentages have been close to share-offamily percentages. The reliability of brand bought as an indicator of share of families was also examined. Data from two additional questions were analyzed: What of do you have in house right now? and particular have you purchased most often in past three weeks? Although there are interesting differences, high similarity in percentages exists across three questions. The top-of-mind awareness data were obtained by asking, When you think of (the type of product) what first comes to mind? This was first question asked a respondent, after she was qualified as principal female shopper and a user of pro uct category. The top-of-mind awareness and share-of-families data for 11 brands are shown in figure. All data points are on a percentage basis. The order is based on market performance accumulated over nine data points. For each change in share-of-families between successive data points was determined algebraically by subtracting share of families for each data point from preceding data point. These values were accumulated and 11 brands were rank-ordered by algebraic total as shown in Column 1 of Tables 1 and 2. Thus, Brand A has largest total data point increase in share of families, and Brand K has largest total decrease in data point share changes.

Outlier detection is a part of data analytics that helps users to find discrepancies in working machines by applying outlier detection algorithm on the captured data for every fixed interval. An outlier is a data point that exhibits different properties from other points due to some external or internal forces. These outliers can be detected by clustering the data points. To detect outliers, optimal clustering of data points is important. The problem that arises quite frequently in statistics is identification of groups or clusters of data within a population or sample. The most widely used procedure to identify clusters in a set of observations is k-means using Euclidean distance. Euclidean distance is not so efficient for finding anomaly in multivariate space. This chapter uses k-means algorithm with Mahalanobis distance metric to capture the variance structure of the clusters followed by the application of extreme value analysis (EVA) algorithm to detect the outliers for detecting rare items, events, or observations that raise suspicions from the majority of the data.

The quality of dataset measured and collected by wireless sensor networks (WSN) is often affected by noise and error that are inherent to resource-constrained sensor nodes. The affected data points deviating from the normal pattern are termed as outlier(s). However, detected outlier can be a result of the occurrence of an actual event. Outlier detection techniques developed for WSNs perform binary labelling of the data points and does not indicate the context stating whether the outlier is the result of an actual event or the noise/error. This paper proposes a contextual outlier detection framework specifically designed for WSNs named as in-network contextual outlier detection on edge (INCODE). The proposed framework also estimates the degree of outlierness associated with the detected outlier(s) to provide better insight into the measured data point. Algorithms used in INCODE are designed around the edge computing concept to minimize the communication and computational complexities to make it suitable for resource-constrained WSN. The results suggest an impressive 98% accuracy in identifying the context of the outlier(s). The low communication and computational complexity suggest INCODE’s suitability for resource constrained WSNs.

Stochastic Gradient Descent (SGD) is widely used to train a machine learning model over large datasets, yet its slow convergence rate can be a bottleneck. As a remarkable family of variance reduction techniques, memory algorithms such as SAG and SAGA have been proposed to accelerate the convergence rate of SGD. However, these algorithms need to store per training data point corrections in memory. The unlimited space usage feature is impractical for modern large-scale applications, especially over data points that arrive over time (referred to as streaming data in this paper). To overcome this weakness, this paper investigates the methods that bound the space usage in the state-of-the-art family of variance-reduced stochastic gradient descent over streaming data, and presents CHEAPS2AGA. At each step of updating the model, the key idea of CHEAPS2AGA is always reserving N random data points as samples, while re-using information about past stochastic gradients across all the observed data points with limited space usage. In addition, training an accurate model over streaming data requires the algorithm to be time-efficient. To accelerate the model training phase, CHEAPS2AGA embraces a lock-free data structure to insert new data points and remove unused data points in parallel, and updates the model parameters without using any locking. We conduct comprehensive experiments to compare CHEAPS2AGA to prior related algorithms suited for streaming data. The experimental results demonstrate the practical competitiveness of CHEAPS2AGA in terms of scalability and accuracy.

Defining outliers by their distance to neighboring data points has been shown to be an effective non-parametric approach to outlier detection. In recent years, many research efforts have looked at developing fast distance-based outlier detection algorithms. Several of the existing distance-based outlier detection algorithms report log-linear time performance as a function of the number of data points on many real low-dimensional datasets. However, these algorithms are unable to deliver the same level of performance on high-dimensional datasets, since their scaling behavior is exponential in the number of dimensions. In this paper, we present RBRP, a fast algorithm for mining distance-based outliers, particularly targeted at high-dimensional datasets. RBRP scales log-linearly as a function of the number of data points and linearly as a function of the number of dimensions. Our empirical evaluation demonstrates that we outperform the state-of-the-art algorithm, often by an order of magnitude.

Preserving the performance of a trained model while removing unique characteristics of marked training data points is challenging. Recent research usually suggests retraining a model from scratch with remaining training data or refining the model by reverting the model optimization on the marked data points. Unfortunately, aside from their computational inefficiency, those approaches inevitably hurt the resulting model's generalization ability since they remove not only unique characteristics but also discard shared (and possibly contributive) information. To address the performance degradation problem, this paper presents a novel approach called Performance Unchanged Model Augmentation (PUMA). The proposed PUMA framework explicitly models the influence of each training data point on the model's generalization ability with respect to various performance criteria. It then complements the negative impact of removing marked data by reweighting the remaining data optimally. To demonstrate the effectiveness of the PUMA framework, we compared it with multiple state-of-the-art data removal techniques in the experiments, where we show the PUMA can effectively and efficiently remove the unique characteristics of marked training data without retraining the model that can 1) fool a membership attack, and 2) resist performance degradation. In addition, as PUMA estimates the data importance during its operation, we show it could serve to debug mislabelled data points more efficiently than existing approaches.

Wireless sensor networks (WSNs) are susceptible to faults in sensor data. Outlier detection is crucial for ensuring the quality of data analysis in WSNs. This paper proposes a novel improved support vector data description method (ID-SVDD) to effectively detect outliers of sensor data. ID-SVDD utilizes the density distribution of data to compensate SVDD. The Parzen-window algorithm is applied to calculate the relative density for each data point in a data set. Meanwhile, we use Mahalanobis distance (MD) to improve the Gaussian function in Parzen-window density estimation. Through combining new relative density weight with SVDD, this approach can efficiently map the data points from sparse space to high-density space. In order to assess the outlier detection performance, the ID-SVDD algorithm was implemented on several datasets. The experimental results demonstrated that ID-SVDD achieved high performance, and could be applied in real water quality monitoring.