Multimode complex process monitoring using double‐level local information based local outlier factor method

The ubiquity of smartphones in our daily lives has made them attractive targets for malicious actors seeking to compromise user data and device functionality. Android malware detection has become imperative to protect user privacy and device integrity. This paper presents a focused study on leveraging the Local Outlier Factor (LOF) method for Android malware detection using the DREBIN dataset. Our research addresses the need for accurate and efficient Android malware detection. We explore the LOF method, an anomaly-based detection technique, to assess its effectiveness in distinguishing malicious applications from benign ones within the Android ecosystem. Rigorous experiments using the extensive DREBIN dataset reveal LOF's superiority in accuracy, precision, recall, and False Positive Rate (FPR). We introduce additional metrics like Area Under the Curve (AUC), Matthews Correlation Coefficient (MCC), and True Negative Rate (TNR) to comprehensively evaluate LOF. Our findings highlight LOF's ability to balance false positives and false negatives, making it an ideal choice for Android malware detection. We emphasize the importance of representative datasets, such as DREBIN, for validation. In conclusion, this research positions LOF as a reliable tool for Android malware detection, offering robust protection against emerging threats. As mobile technology evolves, our study encourages further exploration of advanced techniques and real-world deployment scenarios.

K-Means is the most popular of clustering method, but its drawback is sensitivity to outliers. This paper discusses the addition of the outlier removal method to the K-Means method to improve the performance of clustering. The outlier removal method was added to the Local Outlier Factor (LOF). LOF is the representative outlier’s detection algorithm based on density. In this research, the method is called LOF K-Means. The first applying clustering by using the K-Means method on hotspot data and then finding outliers using the LOF method. The object detected outliers are then removed. Then new centroid for each group is obtained using the K-Means method again. This dataset was taken from the FIRM are provided by the National Aeronautics and Space Administration (NASA). Clustering was done by varying the number of clusters (k = 10, 15, 20, 25, 30, 35, 40, 45 and 50) with cluster optimal is k = 20. The result based on the value of Sum of Squared Error (SSE) shown the LOF K-Means method was better than the K-Means method.

Key principal components with recursive local outlier factor for multimode chemical process monitoring

Outlier detection is the process of detecting the data objects which are grossly different from or inconsistent with the remaining set of data. Some of the important applications in the field of data mining are fraud detection, customer behavior analysis, and intrusion detection. There are number of good research algorithms for detecting outliers if the entire data is available and algorithms can operate in more than single passes to achieve the required results. Among the existing methods, LOF (Local outlier Factor) a density based method is very efficient in detecting all forms of outliers. LOF algorithm can not be directly applied to the datastream as the large number of nearest neighbor searches, LOF computation and lrd (local reachability distances) can make it highly inefficient for datastream. In this paper we propose a cluster based partitioning algorithm which can divide the stream in safe region and candidate regions. In Second phase apply LOF algorithm over these partitions separately with some slight enhancement for LOF computation over candidate region to achieve accurate results for finding most outstanding outliers. Several experiments on different dataset confirm that our technique can find better outliers with low computational cost than the direct LOF or compared to the other enhancements proposed for LOF.

Many local outlier detection algorithms have been proposed inspired by the idea of local outlier factor (LOF). However, they often have low detection performance and are sensitive to neighborhood size because there is a major defect in their calculation formulas of outlier degree and the kNN (k-nearest neighbors) method is widely used to quantify a neighborhood of an instance. To address these issues, we define a novel nearest neighbors tree (NNT) to measure a neighborhood of an instance. Meanwhile, we propose a local structure outlier factor (LSOF), which score each local structure instead of each data point and report the top-scored local structures as anomalous local structures, where outliers and groups of outliers are easily divided according to characteristics of the NNT. Our experimental results demonstrate that the competitive behavior of our method on both synthetic and real-world datasets.

After the local outlier factor was first proposed, there is a large family of local outlier detection approaches derived from it. Since the existing approaches only focus on the extent of overall separation between an object and its neighbors, and ignore the degree of dispersion between them, the precision of these approaches will be affected by various degrees in the scattered datasets. In addition, the outlier data occupy a relatively small amount in the dataset, but the existing approaches need to perform local outlier factor calculation on all data during the outlier detection, which greatly reduces the efficiency of the algorithms. In this paper, we redefine a local outlier factor called local deviation coefficient (LDC) by taking full advantage of the distribution of the object and its neighbors. And then, we propose a safe non-outlier objects elimination approach named as rough clustering based on multi-level queries (RCMLQ) to preprocess the datasets to eliminate the non-outlier objects to the utmost. Finally, an efficient local outlier detection approach named as efficient density-based local outlier detection for scattered data (E2DLOS) is proposed based on the LDC and RCMLQ. The RCMLQ greatly reduces the amount of data that needs to be quantified for local outlier factor and the LDC is more sensitive to the degree of anomaly of the scattered datasets, and so the E2DLOS improves the existing local outlier detection approaches in time efficiency and detection accuracy. Experiments show that the LDC can better reflect the true abnormal situations of the data for the scattered datasets. And the RCMLQ can be used in parallel with the traditional methods of improving the efficiency of the nearest neighbor search, which can further improve the efficiency of the E2DLOS algorithm by about 16%.

Aiming at improving the effectiveness and real‐time performance of the fault detection of the photovoltaic (PV) array under different outdoor conditions, a novel method based on Grubbs criterion and local outlier factor (G‐LOF) method is proposed in this study. The simulation model of the PV array is built to obtain the reference current of each PV string. The local outlier factor (LOF) is a ratio of the average local reachability density of sample neighbourhoods relative to its local reachability density. In this study, the LOF is used to quantify the abnormality by comparing the measured current and the simulated reference current. To reduce the false alarm probability of LOF, the Grubbs criterion is utilised as the test criterion to detect the abnormality of PV strings and it is merged with the LOF to derivate the G‐LOF. The experiments of partial shading, open‐circuit fault and short‐circuit fault of PV modules are implemented in sunny and cloudy days. Experimental results verify that the proposed G‐LOF method can sensitively detect above abnormalities, especially can detect the slight performance reduction caused by the partial shading or faults.

Mode changes in today's sophisticated industrial processes result in unanticipated shutdowns, which could reduce the lifespan of critical machinery and result in high maintenance costs. A technique that can identify the fault of equipment operating in various modes is needed to prevent this issue. In light of this, we suggest a novel fault detection technique that makes use of the k-nearest neighbor normalization-based weight local outlier factor (WLOF). The suggested approach carries out local normalization using neighbors to take into account potential mode changes in the normal data, and WLOF is used for fault detection. In contrast to statistical methods, such as principal component analysis (PCA) and independent component analysis (ICA), the local outlier factor (LOF) uses the density of neighbors. However, conventional LOF is greatly affected by the distance between its neighbors, performance may deteriorate when faulty data is adjacent to normal data. To improve the defect detection performance of LOF, the proposed method multiplies proportionally by the distance between each neighbor. A circulating fluidized bed boiler and a multimode numerical case were used to assess the effectiveness of the suggested method. The results of the experiments demonstrate that the suggested method performs better than conventional PCA, kernel PCA (KPCA), k-nearest neighbor (kNN), and LOF. In particular, compared to conventional techniques, the suggested method increased detection accuracy by 20%. Therefore, the proposed technique is applicable to a real process operating in multiple modes.

To meet the demand for diversified products, industrial processes typically have multiple operating modes. In this article, a novel scheme based on aligned mixture factor analysis (AMFA) is proposed for multimode process monitoring. First, mixture factor analysis (MFA) is applied to produce a statistical fingerprint of the training data that gives a detailed description of the multiple operating modes. Then, unlike conventional multimodal algorithms in which multiple models are constructed and monitoring results are softly combined, the proposed method aims at aligning the separated local models together and performing the monitoring behaviors based on the global model. Through this approach of dividing and integrating, both the within-mode and cross-mode correlations can be greatly preserved in the global model. Finally, the utility and feasibility of the proposed method are demonstrated through a numerical example, a nonisothermal continuous stirred-tank reactor (CSTR) model, and the Tennessee Eastman benchmark process.

In this work, we present the first distributed solution for the Local Outlier Factor (LOF) method -- a popular outlier detection technique shown to be very effective for datasets with skewed distributions. As datasets increase radically in size, highly scalable LOF algorithms leveraging modern distributed infrastructures are required. This poses significant challenges due to the complexity of the LOF definition, and a lack of access to the entire dataset at any individual compute machine. Our solution features a distributed LOF pipeline framework, called DLOF. Each stage of the LOF computation is conducted in a fully distributed fashion by leveraging our invariant observation for intermediate value management. Furthermore, we propose a data assignment strategy which ensures that each machine is self-sufficient in all stages of the LOF pipeline, while minimizing the number of data replicas. Based on the convergence property derived from analyzing this strategy in the context of real world datasets, we introduce a number of data-driven optimization strategies. These strategies not only minimize the computation costs within each stage, but also eliminate unnecessary communication costs by aggressively pushing the LOF computation into the early stages of the DLOF pipeline. Our comprehensive experimental study using both real and synthetic datasets confirms the efficiency and scalability of our approach to terabyte level data.

Industrial processes generally have various operation modes, and fault detection for such processes is important. This paper proposes a method that integrates a variational Bayesian Gaussian mixture model with canonical correlation analysis (VBGMM-CCA) for efficient multimode process monitoring. The proposed VBGMM-CCA method maximizes the advantage of VBGMM in automatic mode identification and the superiority of CCA in local fault detection. First, VBGMM is applied to unlabeled historical process data to determine the number of operation modes and cluster the data in each mode. Second, local CCA models that explore input and output relationships are established. Fault detection residuals are generated in each local CCA model, and monitoring statistics are derived. Finally, a Bayesian inference probability index that integrates monitoring results from all local models is developed to increase the monitoring robustness. The effectiveness of the proposed monitoring scheme is verified through experimental studies on a numerical example and the multiphase batch-fed penicillin fermentation process. Note to Practitioners —Process monitoring is important in guaranteeing process safety and improving product quality. Large amounts of unlabeled process data with multiple operation modes generally exist in industrial processes. Labeling these data is difficult or costly. Hence, this paper presents a VBGMM-CCA method for monitoring multimode processes. The key advantage of the proposed method is that it automatically identifies the number of operation modes in historical data and clusters the data. Then, local CCA monitors are established to model the process input and output relationships. During online monitoring, the running-on operation mode is identified through a density function, and the process status is evaluated by the corresponding CCA monitor. A probabilistic monitoring index is also developed to increase the robustness of the monitoring. In comparison with the results of conventional methods, the monitoring results of the proposed approach are more reliable and informative because the process status and the type of the detected fault are presented.

Visual process monitoring is the application of a visualization method to map the real-time operating information of an industrial process to a 2-D map, followed by process monitoring. However, owing to the complexity of industrial production processes and the complex correlations among industrial process variables, the structure and distribution of high-dimensional industrial data are very complicated. Therefore, a general visualization method cannot effectively separate the different fault data in a 2-D map for process monitoring. Accordingly, in this article, a deep double supervised embedding neural network (DDSE) is proposed for visualizing high-dimensional industrial data. The DDSE consists of two supervised deep neural networks: a deep class centres uniform distribution neural network (DCCUD), and a deep supervised t-stochastic neighbor embedding neural network (DSSNE). The DCCUD maps the high-dimensional industrial data to a new feature space in which the class centres obey a uniform distribution, promoting a good and separable situation for subsequent visualization procedures. The DSSNE then maps these high-dimensional features into a 2-D space. The training of the DDSE can be conducted through pre-training and fine-tuning. A proposed visual process monitoring approach combines the DDSE with the local outlier factor and k-nearest neighbor approaches. The proposed approach is tested on a Tennessee Eastman process, and the results illustrate that the proposed approach outperforms traditional methods in terms of visualization and visual process monitoring.

In industrial process monitoring, it is always a challenging and practical problem to analyze the causes of the system fault by isolating true fault variables from vast amounts of process data. However, the phenomenon of smearing effect occurs by using the traditional contribution analysis-based isolation methods since the defined isolation indices of different variables affect each other. In this paper, a new fault isolation method is proposed based on local outlier factor and improved k-nearest neighbor rule aiming to improve the isolation accuracy. Firstly, the nearest neighbors of each sample are obtained along the direction of a specific variable. Based on the nearest neighbors, the outlier-degree value of the variable is calculated and regarded as the contribution of the variable. Then, the contribution of the variable in all samples are obtained in the same way, among which the maximum one is selected as the isolation threshold value of this variable. During the online monitoring, the contribution of the variable in the newly collected sample is calculated in real time. Once the contribution is greater than the threshold, the variable is judged to be the dominant factor causing the system fault. Two cases on numerical example and Tennessee Eastman process are conducted to evaluate the effectiveness of the proposed method.

An unsupervised machine learning model that uses the mechanism of the Local Outlier Factor to flag and detect ambiguous as well as potentially fraudulent claims in Accidental and Healthcare insurance is proposed in this paper. It entirely automates the manual investigation of claims by claim appraisers in any organization. The ethos of this model is to comprehensively automate and expedite the claim investigation process using certain parameters to aid the claim appraiser’s workload of going through straightforward claims and saving their time to investigate more critical and complex claims. The model flags anomalous claims by comparing them to the model’s threshold, and input parameters and alerts are generated. These alerts generated are then investigated for fraud based on the parameters stated. The model can classify these claims and the cost of billable associated with these claims by reporting an accuracy of 99.5% for the Local Outlier Factor model in comparison with other implemented techniques of Isolation Forest which had an accuracy of only 78.37%. Our model has been tested and validated on real-world data and is showing promising results. Being able to identify and flag potentially fraudulent claims before they are paid out can save insurance companies a lot of money and resources.

Outlier detection is an important task in data mining with numerous applications, including credit card fraud detection, video surveillance, etc. A recent work on outlier detection has introduced a novel notion of local outlier in which the degree to which an object is outlying is dependent on the density of its local neighborhood, and each object can be assigned a Local Outlier Factor (LOF) which represents the likelihood of that object being an outlier. Although the concept of local outliers is a useful one, the computation of LOF values for every data objects requires a large number of κ-nearest neighbors searches and can be computationally expensive. Since most objects are usually not outliers, it is useful to provide users with the option of finding only n most outstanding local outliers, i.e., the top-n data objects which are most likely to be local outliers according to their LOFs. However, if the pruning is not done carefully, finding top-n outliers could result in the same amount of computation as finding LOF for all objects. In this paper, we propose a novel method to efficiently find the top-n local outliers in large databases. The concept of "micro-cluster" is introduced to compress the data. An efficient micro-cluster-based local outlier mining algorithm is designed based on this concept. As our algorithm can be adversely affected by the overlapping in the micro-clusters, we proposed a meaningful cut-plane solution for overlapping data. The formal analysis and experiments show that this method can achieve good performance in finding the most outstanding local outliers.