An Outlier Fuzzy Detection Method Using Fuzzy Set Theory

The prevalence of streaming data across various sectors poses significant challenges for real-time anomaly detection due to its volume, velocity, and variability. Traditional data processing methods often need to be improved for such dynamic environments, necessitating robust, scalable, and efficient real-time analysis systems. This study compares two advanced machine learning approaches—LSTM autoencoders and Matrix Profile algorithms—to identify the most effective method for anomaly detection in streaming environments using the NYC taxi dataset. Existing literature on anomaly detection in streaming data highlights various methodologies, including statistical tests, window-based techniques, and machine learning models. Traditional methods like the Generalized ESD test have been adapted for streaming data but often require a full historical dataset to function effectively. In contrast, machine learning approaches, particularly those using LSTM networks, are noted for their ability to learn complex patterns and dependencies, offering promising results in real-time applications. In a comparative analysis, LSTM autoencoders significantly outperformed other methods, achieving an F1-score of 0.22 for anomaly detection, notably higher than other techniques. This model demonstrated superior capability in capturing temporal dependencies and complex data patterns, making it highly effective for the dynamic and varied data in the NYC taxi dataset. The LSTM autoencoder's advanced pattern recognition and anomaly detection capabilities confirm its suitability for complex, high-velocity streaming data environments. Future research should explore the integration of LSTM autoencoders with other machine-learning techniques to enhance further the accuracy, scalability, and efficiency of anomaly detection systems. This study advances our understanding of scalable machine-learning approaches and underscores the critical importance of selecting appropriate models based on the specific characteristics and challenges of the data involved.

In many computer vision applications for recognition or classification, outlier detection plays an important role as it affects the accuracy and reliability of the result. We propose a novel approach for outlier detection using Gaussian process classification. With this approach, the outlier detection can be integrated to the classification process, instead of being treated separately. Experimental results on handwritten digit image recognition and vision based robot localization show that our approach performs better than other state of the art approaches.

A new particle-swarm-optimization-assisted deep transfer learning framework with applications to outlier detection in additive manufacturing

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

Outliers are the data objects or data elements which are deviated from the observation or population in the dataset. Normally outliers are considered as noise data but in recent days outliers are taken vital role in the applications like healthcare systems to energetically check for any malignant, irregular, or abnormal behaviour. Accuracy of the outlier detection is purely based upon the efficiency of outlier detection methods and application where the outlier detection is involved. In this paper we design hybrid three phase modified fuzzy c-means and diverse distance based outlier detection method for distributed dataset to detect the unusual usage of data and illegitimate approach in large-scale integral networks, especially in healthcare sectors (EPR systems). The proposed algorithm combining the features of modified C-Means Fuzzification, Z-score and Manhattan distance in outlier identification. The proposed algorithm provides efficiency in outlier detection on univariate EPRdata. This paper also mainly focuses the application intrusion detection in healthcare data. The algorithm is tested on real world dataset of machine learning database repository (UCIML).

Since the Local Outlier Factor (LOF) was first proposed, there is a large family of approaches that is derived from it. For the reason that the existing local outliers detection approaches only focus on the extent of overall separation between an object and its neighbors, and do not pay attention to the degree of dispersion between them, the precision of these approaches will be seriously affected in the scattered data sets for outlier detection. In this paper, we redefine the local outliers by combining the degree of dispersion of the object and its neighbors, and propose a new local outlier detection approach (N2DLOF). Compared to conventional approaches, the outliers obtained by N2DLOF are more sensitive to the degree of anomaly of the scattered data sets. Experiments show that our approach has a significant improvement on outlier detection precision in the case of scattered datasets with similar time complexity. In short, we extend the ecosystem of the local outlier detection approaches from a new perspective.

Internet of things (IoT) becomes the most popular term in the recent advances in Healthcare devices. The healthcare data in the IoT process and structure is very sensitive and critical in terms of healthy and technical considerations. Outlier detection approaches are considered as principal tool or stage of any IoT system and are mainly categorized in statistical and probabilistic, clustering and classification-based outlier detection. Recently, fuzzy logic (FL) system is used in ensemble and cascade systems with other ML-based tools to enhance outlier detection performance but its limitation involves the false detection of outliers. In this paper, we propose a fuzzy logic system that uses the anomaly score of each point using local outlier factor (LOF), connectivity-based outlier factor (COF) and generalized LOF to eliminate the confusion in classifying points as outliers or inliers. Regarding human activity recognition (HAR) dataset, the FL achieved a value of 98.2 %. Compared to the performance of LOF, COF, and GLOF individually, the accuracy increased slightly, but the increase in precision and recall indicates an increase in correctly classified data and that neither true nor abnormal data is classified wrongly. The results show the increase in precision and recall which indicates an increase in correctly classified data. Thus, it can be confirmed that fuzzy logic with input of scores achieved the desired goal in terms of mitigating cases of false detection of anomalous data. By comparing the proposed ensemble of fuzzy logic and different types of local density scores in this study, the outcomes of fuzzy logic presents a new way of elaborating or fusing the different tools of the same purpose to enhance detection performance

Outlier detection is an important task for fitting a model to a set of data. Two different outlier detection approaches are given as tests for outliers and robust methods. For these approaches, usually outliers are considered as additive bias terms neglected in the original adjustment model. However, there is another approach that outlier is considered as a model error in the Gauss–Markov model. This model error is represented as an unknown parameter. As it cannot be known before which observation includes outlier; this method is applied on the data for each observation separately and tested with t-test or F-test. It is successful if the sample includes only one outlier. To detect multiple outliers more successfully, in this article, a new outlier detection method is introduced. In this method, all the possible combinations of multiple outliers are considered as model errors and it is accepted that the smallest variance of them gives the solution for a certain number of outliers, then the estimated model errors are detected by comparing with a critical value. The critical value is chosen as 3σo. To compare the results of the new method, with those of the Least Median of Squares (LMS) and Huber M-estimators, Monte Carlo simulation technique is used for linear regression. The Mean Success Rate is proposed to measure the reliabilities of the methods. We showed that the new method is robust and includes the property of high breakdown point as LMS; and more efficient than LMS.

Graph clustering and graph outlier detection have been studied extensively on plain graphs, with various applications. Recently, algorithms have been extended to graphs with attributes as often observed in the real-world. However, all of these techniques fail to incorporate the user preference into graph mining, and thus, lack the ability to steer algorithms to more interesting parts of the attributed graph. In this work, we overcome this limitation and introduce a novel user-oriented approach for mining attributed graphs. The key aspect of our approach is to infer user preference by the so-called focus attributes through a set of user-provided exemplar nodes. In this new problem setting, clusters and outliers are then simultaneously mined according to this user preference. Specifically, our FocusCO algorithm identifies the focus, extracts focused clusters and detects outliers. Moreover, FocusCO scales well with graph size, since we perform a local clustering of interest to the user rather than global partitioning of the entire graph. We show the effectiveness and scalability of our method on synthetic and real-world graphs, as compared to both existing graph clustering and outlier detection approaches.

Researchers in science and engineering face various obstacles due to a lack of specific and full data. Many different approaches have been devised to deal with these restrictive requirements, but two notable schools of thought are the fuzzy set (FS) theory and the rough set (RS) theory, both of which have spawned many extensions and hybridizations. Although RS theory originated from an indiscernibility relation (also known as an equivalence relation in mathematics), emphasis rapidly shifted to similarity or coverings (and their fuzzy analogues). Many other hybrid schemes were suggested with this goal in mind. The gap between those concepts shrank because to this thorough analysis. Fuzzy set theory is a legitimate way to convey the ambiguity of assessment data, yet it is still inadequate for dealing with certain intricate problems in the actual world. In reality, decision makers will undoubtedly provide different kinds of ambiguous and nuanced assessments. Atanassov’s intuitionistic fuzzy set theory broadened the application of fuzzy set theory by imbuing it with an element of uncertainty. Sometimes in real life, you have to deal with a neutral element on top of the indeterminate one. Picture fuzzy sets were developed specifically for this purpose. Membership roles may be positive, neutral, or negative/refusal. In contrast, hesitant fuzzy sets and its hybrid models are useful when decision makers are on the fence about which option to choose. As a binary relation on a set, a graph is symmetric. It is a staple in mathematical modelling and is used in almost every scientific and technological discipline. Graph theory has been essential in the mathematical modelling and subsequent resolution of several real-world situations. Information about connections between things is often best represented using graph theory, which uses vertices to stand in for the items and edges for the relationships between them. The suggested dynamic algorithm is better to the static approach in dealing with the multidimensional dynamic changes of the hybrid incomplete decision system, according to a series of experiments carried out on nine UCI datasets.

Since the nonstationary distribution of the detected objects is general in the real world, the accurate and efficient outlier detection for data analysis within wireless sensor network (WSN) is a challenge. Recently, with high classification precision and affordable complexity, one-class quarter-sphere support vector machine (QSSVM) has been introduced to deal with the online and adaptive outlier detection in WSN. Regarding the one-sided consideration of optimization or iterative updating algorithm for QSSVM model within current techniques, we have proposed comprehensive outlier detection methods in WSN based on the QSSVM algorithm. To reduce the complexity of optimization algorithm for QSSVM model in existing techniques, a fast optimization algorithm based on average Euclidean distance has been developed and employed to the comprehensive outlier detection method. Evaluated by real and synthetic WSN data sets, our methods have shown an excellent outlier detection performance, and they have been proved to meet the requirements of online adaptive outlier detection in the case of nonstationary detection tasks of WSN.

The curse of dimensionality in high-dimensional data makes it difficult to capture the abnormality of data points in full data space. To deal with this problem, we propose an outlier detection model based on Variational Autoencoder and Genetic Algorithm for subspace outlier analysis of high-dimensional data (VAGA). The proposed VAGA model constructs a variational autoencoder (VAE) to preliminarily detect outliers. Then the genetic algorithm (GA) is used to search the abnormal subspace of the outliers obtained by the VAE layer to provide a basis for subspace outlier analysis. The subsequent clustering of the abnormal subspaces help filter out the false positives which are fed back to the VAE layer to adjust network weights. The comparative experiments performed on three public benchmark datasets show that the outlier detection results of the proposed VAGA model are highly interpretable and have better accuracy performance than the state-of-the-art outlier detection methods.

Regarding the magnificent ability of the fuzzy logic in explanation and analyzing the real phenomena, using fuzzy data is growing rapidly. But the tools for doing data mining in a fuzzy data set; especially in high dimension, are not sufficient. Considering the remarkable ability of the LoOP (Local Outlier Probabilities) method in outlier detecting in high dimensional crisp data set, like supplying for each data object, determining outlier probability as scar which is easily explainable competitor on various synthetic and real world data sets, the aim of this article is to present an extended version of this method for using in high dimensional fuzzy database. In order to provide a completely fuzzy strategy, first we propose a new approach to assigning distance between fuzzy high dimension data based on Hausdorff and Vertex distance. After proving that this distance measuring is a metric, since LoOP is a distance based method, we redefined the LoOP algorithm step by step based on the new meter. Then we allocate an outlier score in the range of [0,1] to each data based on LoOP idea, to identify the outliers. The presented numerical tests show the efficiency of the new method, for random cases of knowing and unknowing the outlier and a real case. The results shows also the stability and robustness of the presented method more that its accuracy.

Due to the globalization impacts modern manufacturing environments are inevitably complicating and evolving, which is random fuzzy in nature. Therefore, the mathematical treatments in statistical control charts have to deal with the coexisting random and fuzzy uncertainties. Fuzzy mathematics created by Zadeh [5], [6] is possibility measure theory based and the treatments are at fuzzy set level which is difficult to handle. The new fuzzy theory founded by Liu [2] is axiomatic based which convert fuzzy set variable into a scalar fuzzy variable and thus makes the fuzzy mathematical treatments behave similar to those in standard probability theory. In this paper, we introduce the new fuzzy theory-credibility measure theory into the analysis of the average run length (abbreviated as ARL) analysis of Shewhart style statistical control chart under normal fuzzy manufacture environments.

Exploiting fuzzy rough entropy to detect anomalies