Anomaly Detection Algorithm Based on CFSFDP

The problem of anomaly detection in marine Argo data is studied. Based on the common and widely used DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm for anomaly detection in Argo data, as in other fields, there is a major problem in the application of DBSCAN algorithm, which is how to choose the appropriate parameter pairs. To solve this problem, this paper proposes an improved version of DBSCAN, namely CCMD-DBSCAN (DBSCAN based on the classification characteristics of marine data), which solves the above problem by studying the characteristics and laws of Argo data and is successfully applied to anomaly detection of marine data. The experimental results show that the new algorithm can not only determine the appropriate parameter pairs but also has a good anomaly detection effect.KeywordsArgoDBSCANAnomaly detection

Clustering algorithm is one of the most important algorithms in unsupervised learning. For density‐based spatial clustering of applications with noise (DBSCAN) density clustering algorithm, the selection of neighborhood radius and minimum number is the key to get the best clustering results. Aiming at the problems of traditional DBSCAN algorithm, such as the neighborhood radius and the minimum number of points, this article puts forward two classifications based on K‐means algorithm, and gets two clustering centers. Where calculated between two data points and the cluster center‐to ‐center distance, clustering, distance, statistics in a distance of data points within the scope of the search, the number of data points corresponding to the maximum distance value, and thus the parameters for the DBSCAN algorithm to estimate and selection of initial radius of neighborhood with the minimum number of clustering start critical value. When the parameters are iterated and optimized continuously, the data are divided into clusters, and the most suitable neighborhood radius and the minimum point number are obtained. The experimental data analysis show that the improved algorithm reduces the human factors in the traditional algorithm and improves the efficiency, so as to get the accurate clustering results.

Crowdsourced drone LiDAR data, collected from diverse sensors and environments, often suffer from inconsistent data quality.Outliers in such datasets can distort the characteristics of terrain and structures, leading to inaccurate decision-making.This study aims to validate the quality of crowdsourced LiDAR data by developing a precise outlier detection method based on semantic segmentation.Using the open-source Semantic Terrain Points Labeling Synthetic 3D (STPLS3D) dataset, noise-augmented training data were generated through simulations.Subsequently, a Kernel Point Convolution (KPConv) model was trained.The trained model was applied to real-world crowdsourced LiDAR data and compared with existing methods, including Statistical Outlier Removal (SOR) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN).Experimental results demonstrated that the KPConv model outperformed SOR and DBSCAN in terms of accuracy and reliability, effectively identifying inherent noise in the original data.These findings highlight the utility of deep learning-based outlier detection methods for ensuring the quality of crowdsourced LiDAR data.

Anomaly Detection is an important area of research in data science. However, for big data it is challenging due to limited (or no) labeled data during training. This challenge is prevalent in the cyber-security area. There is always a lag before labels of (near) real time data become available. Hence, we would like to develop a new anomaly detection framework without any labeled data (no human intervention). For this, we utilize the density-based spatial clustering of applications with noise (DBSCAN). To address the scalability issues that arise when using DBSCAN, we exploit novel techniques based on partitions with minimum false alarm. We compare our approaches with other unsupervised approaches to demonstrate the effectiveness of our work.

Density-Based Spatial Clustering of Application with Noise (DBSCAN) is a well-known, unsupervised machine learning tool for clustering. DBSCAN creates clusters based on dense regions while marking points that lie alone in low-density regions as outliers. In DBSCAN, the density is detected via core points which are quite sensitive to input parameters: ϵ is radius of the neighborhood and MinPts is minimum number of points constraint within ϵ radius. In contrast to crisp core point definition, intuitionistic fuzzy core point definition makes DBSCAN algorithm capable to detect different patterns of density by two different combinations of input parameters. In this study, a DBSCAN extension is proposed based on this idea: IFDBSCAN. The proposed extension is then illustrated by a computational experiment on a synthetic dataset. Results show that, IFDBSCAN can establish fine-tuned clusters relatively to classical DBSCAN and provide users more insight on the estimation of input parameters.

As the foundation of global communication networks, optical fibers are vulnerable to various disruptive events, including mechanical damage, such as cuts, and malicious physical layer breaches, such as eavesdropping via fiber bending. Traditional monitoring methods often fail to identify subtle or novel anomalies, stimulating the proliferation of Machine Learning (ML) techniques for detection of threats before they cause significant harm. In this paper, we evaluate the performance of Semi-Supervised Learning (SSL) and Unsupervised Learning (USL) approaches for detecting various abnormal events, such as fiber bending and vibrations, by analyzing polarization signatures with minimal reliance on labeled data. We experimentally collect thirteen polarization signatures on three different types of fiber cable and process them using One-Class Support Vector Machine (OCSVM) as an SSL, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) as a USL algorithm for anomaly detection. We introduce tailored evaluation metrics designed to guide hyper-parameter tuning and capture generalization over different anomaly types, detection consistency, and robustness to false positives, enabling practical deployment of OCSVM and DBSCAN in optical fiber security. Our findings demonstrate DBSCAN as a strong contender to detect previously unseen threats in scenarios where labeled data are not available, despite some variability in performance between different scenarios, with F1 score values between 0.615 and 0.995. In contrast, OCSVM, trained on normal operating conditions, maintains high F1 scores of 0.98 to 0.998, demonstrating accurate detection of complex anomalies in optical networks.

PurposeRecent investigations failed to reproduce the positive rotor-guided ablation outcomes shown by initial studies for treating persistent atrial fibrillation (persAF). Phase singularity (PS) is an important feature for AF driver detection, but algorithms for automated PS identification differ. We aim to investigate the performance of four different techniques for automated PS detection.Methods2048-channel virtual electrogram (VEGM) and electrocardiogram signals were collected for 30 s from 10 patients undergoing persAF ablation. QRST-subtraction was performed and VEGMs were processed using sinusoidal wavelet reconstruction. The phase was obtained using Hilbert transform. PSs were detected using four algorithms: (1) 2D image processing based and neighbor-indexing algorithm; (2) 3D neighbor-indexing algorithm; (3) 2D kernel convolutional algorithm estimating topological charge; (4) topological charge estimation on 3D mesh. PS annotations were compared using the structural similarity index (SSIM) and Pearson’s correlation coefficient (CORR). Optimized parameters to improve detection accuracy were found for all four algorithms using Fβ score and 10-fold cross-validation compared with manual annotation. Local clustering with density-based spatial clustering of applications with noise (DBSCAN) was proposed to improve algorithms 3 and 4.ResultsThe PS density maps created by each algorithm with default parameters were poorly correlated. Phase gradient threshold and search radius (or kernels) were shown to affect PS detections. The processing times for the algorithms were significantly different (p < 0.0001). The Fβ scores for algorithms 1, 2, 3, 3 + DBSCAN, 4 and 4 + DBSCAN were 0.547, 0.645, 0.742, 0.828, 0.656, and 0.831. Algorithm 4 + DBSCAN achieved the best classification performance with acceptable processing time (2.0 ± 0.3 s).ConclusionAF driver identification is dependent on the PS detection algorithms and their parameters, which could explain some of the inconsistencies in rotor-guided ablation outcomes in different studies. For 3D triangulated meshes, algorithm 4 + DBSCAN with optimal parameters was the best solution for real-time, automated PS detection due to accuracy and speed. Similarly, algorithm 3 + DBSCAN with optimal parameters is preferred for uniform 2D meshes. Such algorithms – and parameters – should be preferred in future clinical studies for identifying AF drivers and minimizing methodological heterogeneities. This would facilitate comparisons in rotor-guided ablation outcomes in future works.

Background: This study focuses on the latest knowledge regarding Micro, Small and Medium Enterprises (MSMEs) as a current central issue. These enterprises have shown their significance in providing employment opportunities and contributing to the country’s economy. However, MSMEs face various challenges that must be addressed to optimize their outcomes. Understanding the characteristics of this group was crucial in formulating effective strategies. Objective: This study proposed to cluster or combine micro, small, and medium enterprises (MSMEs) data in a particular area based on asset value and turnover. As a result, this study aimed to gain insights into the MSME landscape in the area and provided valuable information for decision-makers and stakeholders. Methods: This study utilized two methods, namely the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method and the K-Means method. These methods were chosen for their distinct capabilities. DBSCAN was selected for its ability to handle noisy data and identify clusters with diverse forms, while K-Means was chosen for its popularity and ability to group data based on proximity. The study used a dataset containing MSME information, including asset values and turnover, collected from various sources. Results: The outcomes encompassed identifying clusters of MSMEs based on their closeness in the feature space within a specific region. Optimizing the clustering outcomes involved modifying algorithm parameters like epsilon and minimum points for DBSCAN and the number of clusters for K-Means. Furthermore, this study attained a deeper understanding of the arrangement and characteristics of MSME clusters in the region through a comparative analysis of the two methodologies. Conclusion: This study offered perspectives on clustering MSMEs based on asset value and turnover in a specific region. Employing DBSCAN and K-Means methodologies allowed researchers to depict the MSME landscape and grasp the business attributes of these enterprises. These results could aid in decision-making and strategic planning concerning the advancement of the MSME sector in the mentioned area. Future study may investigate supplementary factors and variables to deepen comprehension of MSME clusters and promote regional growth and sustainability. Keywords: Asset Value, Clustering, DBSCAN, K-Means, Turnover

The density based notion for clustering approach is used widely due to its easy implementation and ability to detect arbitrary shaped clusters in the presence of noisy data points without requiring prior knowledge of the number of clusters to be identified. Density-based spatial clustering of applications with noise (DBSCAN) is the first algorithm proposed in the literature that uses density based notion for cluster detection. Since most of the real data set, today contains feature space of adjacent nested clusters, clearly DBSCAN is not suitable to detect variable adjacent density clusters due to the use of global density parameter neighborhood radius Nrad and minimum number of points in neighborhood Npts. So the efficiency of DBSCAN depends on these initial parameter settings, for DBSCAN to work properly, the neighborhood radius must be less than the distance between two clusters otherwise algorithm merges two clusters and detects them as a single cluster. Through this paper: 1) We have proposed improved version of DBSCAN algorithm to detect clusters of varying density adjacent clusters by using the concept of neighborhood difference and using the notion of density based approach without introducing much additional computational complexity to original DBSCAN algorithm. 2) We validated our experimental results using one of our authors recently proposed space density indexing (SDI) internal cluster measure to demonstrate the quality of proposed clustering method. Also our experimental results suggested that proposed method is effective in detecting variable density adjacent nested clusters.

An efficient and scalable density-based clustering algorithm for datasets with complex structures

Over the last two decades, anomaly detection (AD) has been known to play a critical role in hyperspectral image analysis, which provides a new way to distinguish the targets from the background without prior knowledge. Recently, the representation-based methods were proposed and soon became a significant type of methods on hyperspectral AD. In this paper, a novel AD algorithm based on convolutional neural network (CNN) and low-rank representation (LRR) is proposed. First, a CNN model is built and trained on hyperspectral image (HSI) datasets to accurately obtain the resulting abundance maps. Compared with the raw dataset, abundance maps contain more distinctive features to identify anomalies from the background. Second, a dictionary is constructed by the density-based spatial clustering of applications with noise (DBSCAN) algorithm to stably represent the background component. Third, a matrix decomposition method based on LRR is adopted. In this way, a coefficient matrix corresponding to the constructed dictionary is obtained, which is low rank. At the same time, a residual matrix can be obtained as well, which is sparse. Finally, anomalies can be extracted from the residual matrix. The experimental results show that the proposed method achieves a superior performance compared to some of the state-of-the-art methods in the field of hyperspectral AD.

Like other mining, web mining is also necessary to increase the power of web search engine to identify the intended web page and web document. While processing with large datasets, there arises several issues associated with space availability, similarity relationships between different webpage’s and running time. Hence, this paper intends to develop an enhanced web mining model based on two contributions. At first, the hierarchical tree is framed, which produces different categories of the searching queries (different web pages). Next, to hierarchical tree model, enhanced Density-Based Spatial Clustering of Applications with Noise (DBSCAN) technique model is developed by modifying the traditional DBSCAN. This technique results in proper session identification from raw data. Moreover, this technique offers the optimal level of clusters necessitated for hierarchical clustering. After hierarchical clustering, the rule mining is adopted. The traditional rule mining technique is generally based on the frequency; however, this paper intends to enhance the traditional rule mining based on utility factor as the second contribution. Hence the proposed model for web rule mining is termed as Enhanced DBSCAN-based Hierarchical Tree (EDBHT). It benefits in providing the search results depending on high level information (e.g., location), so that the ability of search engine in providing the interesting association rules can be improved. Next, to the implementation, the performance of proposed EDBHT is found to be enhanced when compared over several traditional models.

This study proposes a novel robust unit commitment (UC) framework with data-driven disjunctive uncertainty sets for volatile wind power generation, which integrates a two-stage adaptive robust UC model with machine learning techniques to flexibly capture the uncertainty space of the wind power forecast errors with disjunctive structures. K-means clustering and density-based spatial clustering of applications with noise (DBSCAN) are applied for clustering, and data-driven disjunctive uncertainty sets are constructed as the union of multiple basic uncertainty sets following the clustering results. The problem is formulated into a two-stage adaptive robust UC model with data-driven disjunctive uncertainty sets and with a multi-level optimization structure. A tailored decomposition-based optimization algorithm is developed to facilitate the solution process. A numerical experiment and a UC case study on the IEEE 39-bus system are presented to demonstrate the effectiveness of the proposed approach. In both applications, using the proposed disjunctive uncertainty sets can effectively reduce the price of robustness compared to the conventional "one-set-fits-all" approach. Furthermore, the disjunctive uncertainty sets using DBSCAN tend to provide less conservative solutions than those using K-means, implying that DBSCAN can handle the outliers and noise of the uncertainty data more efficiently.

Revised DBSCAN algorithm to cluster data with dense adjacent clusters

Wind farm power data is an essential data source for characteristics analysis of wind farms, and its precision highly influences the operation and control of wind power. Existing methods for abnormal wind power data detection are mainly based on supervised or unsupervised algorithms, which may suffer from huge effort consumption on artificial label-setting or improvable detection precision respectively. In this paper, a semi-supervised anomaly detection method based on Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is proposed. Supervision from a limited number of labelled data is utilized to guide the anomaly detection process. Simulation based on realistic wind farm power data from China is performed to verify the method validity. The simulation results illustrate that the proposed method can obtain a satisfactory performance on avoiding false dismissals and false alarms among irregular-shaped wind power clusters, as well as an improved distinguishing ability around the confusing boundaries between normal and abnormal data.