Enhanced Stratified Sampling-Density-Based Spatial Clustering of Applications With Noise (SS-DBSCAN) for High-Dimensional Data

Smart meters have become a core part of the Internet of Things, and its sensory network is increasing globally. For example, in the UK there are over 15 million smart meters operating across homes and businesses. One of the main advantages of the smart meter installation is the link to a reduction in carbon emissions. Research shows that, when provided with accurate and real-time energy usage readings, consumers are more likely to turn off unneeded appliances and change other behavioural patterns around the home (e.g., lighting, thermostat adjustments). In addition, the smart meter rollout results in a lessening in the number of vehicle callouts for the collection of consumption readings from analogue meters and a general promotion of renewable sources of energy supply. Capturing and mining the data from this fully maintained (and highly accurate) sensing network, provides a wealth of information for utility companies and data scientists to promote applications that can further support a reduction in energy usage. This research focuses on modelling trends in domestic energy consumption using density-based classifiers. The technique estimates the volume of outliers (e.g., high periods of anomalous energy consumption) within a social class grouping. To achieve this, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Ordering Points to Identify the Clustering Structure (OPTICS) and Local Outlier Factor (LOF) demonstrate the detection of unusual energy consumption within naturally occurring groups with similar characteristics. Using DBSCAN and OPTICS, 53 and 208 outliers were detected respectively; with 218 using LOF, on a dataset comprised of 1,058,534 readings from 1026 homes.

Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) shows great potential for forest height retrieval. However, there are abundant noise photons in the ICESat-2 data, which make the accurate extraction of global forest heights challenging. In this letter, a novel algorithm based on the clustering method of ordering points to identify the clustering structure (OPTICS) was proposed to remove noise photons. First, we modified the circular shape of the search area in the OPTICS algorithm to an elliptical shape. Second, a distance ordering of all photons was generated using the modified OPTICS algorithm. Finally, signal photons were effectively detected using distance thresholds set by the Otsu method. To evaluate the algorithm performance, both the simulated and real ICESat-2 data were applied to our proposed algorithm. In addition, we compared our algorithm with another noise removal algorithm based on the modified density-based spatial clustering of applications with noise (DBSCAN). The results show that our algorithm works well in distinguishing the signal and noise photons as indicated by high <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F$ </tex-math></inline-formula> values. Compared with the modified DBSCAN, our algorithm performs better in filtering out noise photons regardless of the simulated or real ICESat-2 data sets. In addition, the results also indicate that our algorithm is robust because it is insensitive to the clustering parameters. Overall, the new proposed algorithm is effective for removing noise photons in the ICESat-2 data.

Density-Based Spatial Clustering of Application with Noise (DBSCAN) is a well-known, unsupervised machine learning tool for clustering. DBSCAN creates clusters based on dense regions while marking points that lie alone in low-density regions as outliers. In DBSCAN, the density is detected via core points which are quite sensitive to input parameters: ϵ is radius of the neighborhood and MinPts is minimum number of points constraint within ϵ radius. In contrast to crisp core point definition, intuitionistic fuzzy core point definition makes DBSCAN algorithm capable to detect different patterns of density by two different combinations of input parameters. In this study, a DBSCAN extension is proposed based on this idea: IFDBSCAN. The proposed extension is then illustrated by a computational experiment on a synthetic dataset. Results show that, IFDBSCAN can establish fine-tuned clusters relatively to classical DBSCAN and provide users more insight on the estimation of input parameters.

The interplay between political campaigning and online discourse has emerged as a critical area of computational social science, particularly concerning the propagation of hate speech. This study investigates how Donald Trump’s 2024–2025 U.S. presidential campaign influenced the prevalence and thematic focus of hate speech on Reddit, a platform known for its politically engaged communities. Leveraging a dataset of over 55 million Reddit posts spanning two key periods (pre- and mid-campaign), we develop a large-scale Natural Language Processing (NLP) methodology to detect and analyse the dynamics of hate speech. Hate speech is identified using a fine-tuned BERT model optimized for multi-class classification, distinguishing hateful, offensive, and neutral content. To explore the structure of hate speech targets, we apply Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), a density-based clustering algorithm that accommodates variable data distributions in high-dimensional embeddings. A post-processing algorithm further refines overlapping categories by decomposing identity-based slurs into distinct subgroups. Our findings reveal a marked increase in hate speech volume during the campaign, along with significant shifts in targeted identity clusters, particularly toward ideological, gendered, and racial groups. These results contribute empirical evidence to debates around algorithmic content moderation, political communication, and the computational modeling of social polarization.

We explored the use of broadband colors to classify stars, galaxies, and quasi-stellar objects (QSOs). Specifically, we applied sharpened dimensionality reduction (SDR)-aided classification to this problem, with the aim of enhancing cluster separation in the projections of high-dimensional data clusters to allow for better classification performance and more informative projections. The main objective of this work was to apply SDR to large sets of broadband colors derived from the CPz catalog to obtain projections with clusters of star, galaxy, and QSO data that exhibit a high degree of separation. The SDR method achieves this by combining density-based clustering with conventional dimensionality-reduction techniques. To make SDR scalable and have the ability to project samples using the earlier-computed projection, we used a deep neural network trained to reproduce the SDR projections. Subsequently classification was done by applying a $k$-nearest neighbors ($k$-NN) classifier to the sharpened projections. Based on a qualitative and quantitative analysis of the embeddings produced by SDR, we find that SDR consistently produces accurate projections with a high degree of cluster separation. A number of projection performance metrics are used to evaluate this separation, including the trustworthiness, continuity, Shepard goodness, and distribution consistency metrics. Using the $k$-NN classifier and consolidating the results of various data sets, we obtain precisions of 99.7&lt;!PCT!&gt;, 98.9&lt;!PCT!&gt;, and 98.5&lt;!PCT!&gt; for classifying stars, galaxies, and QSOs, respectively. Furthermore, we achieve completenesses of 97.8&lt;!PCT!&gt;, 99.3&lt;!PCT!&gt;, and 86.8&lt;!PCT!&gt;, respectively. In addition to classification, we explore the structure of the embeddings produced by SDR by cross-matching with data from Gaia DR3, Galaxy Zoo 1, and a catalog of specific star formation rates, stellar masses, and dust luminosities. We discover that the embeddings reveal astrophysical information, which allows one to understand the structure of the high-dimensional broadband color data in greater detail. We find that SDR-aided star, galaxy, and QSO classification performs comparably to another unsupervised learning method using hierarchical density-based spatial clustering of applications with noise (HDBSCAN) but offers advantages in terms of scalability and interpretability. Furthermore, it outperforms traditional color selection methods in terms of QSO classification performance. Overall, we demonstrate the potential of SDR-aided classification to provide an accurate and physically insightful classification of astronomical objects based on their broadband colors.

Image segmentation is the process to divide a digital image into a number of regions for further usage. Color images can be segmented by applying density-based clustering methods, e.g. Density-Based Spatial Clustering of Applications with Noise (DBSCAN), which is used to identify the arbitrary shaped clusters. The drawback of DBSCAN is the high computational complexity whilst the size of image input is normally very large. Self-Organizing Map (SOM) is a dimensionality reduction method which can be applied to reduce the dimensions of image processing tasks. This paper proposes a hybrid method of SOM and DBSCAN (SOM-DBSCAN) for image segmentation. To evaluating the usability of the proposed SOM-DBSCAN method, four images are used to benchmark image segmentation.

Arecaceae (palms) are an important resource for indigenous communities as well as fauna populations across Amazonia. Understanding the spatial patterns and the environmental factors that determine the habitats of palms is of considerable interest to rainforest ecologists. Here, we utilize remotely sensed imagery in conjunction with topography and soil attribute data and employ a generalized cluster identification algorithm, Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), to study the underlying patterns of palms in two areas of Guyana, South America. The results of the HDBSCAN assessment were cross-validated with several point pattern analysis methods commonly used by ecologists (the quadrat test for complete spatial randomness, Morista Index, Ripley’s L-function, and the pair correlation function). A spatial logistic regression model was generated to understand the multivariate environmental influences driving the placement of cluster and outlier palms. Our results showed that palms are strongly clustered in the areas of interest and that the HDBSCAN’s clustering output correlates well with traditional analytical methods. The environmental factors influencing palm clusters or outliers, as determined by logistic regression, exhibit qualitative similarities to those identified in conventional ground-based palm surveys. These findings are promising for prospective research aiming to integrate remote flora identification techniques with traditional data collection studies.

&lt;span lang="EN-US"&gt;Automatic identification system (AIS) is a vessel radio navigation equipment that has been determined by international maritime organization (IMO). Historical AIS data can be utilized for anomaly detection, trajectory prediction, and vessel trajectory planning. These benefits can be achieved by identifying the vessel's trajectory pattern through trajectory clustering. However, more effort is needed in trajectory clustering using AIS data due to their large volume and the significant number of deficiencies. In addition, trajectory clustering cannot be directly applied to trajectory data, which also applies to vessel trajectory. Therefore, we propose a trajectory clustering framework by combining douglas peucker (DP), longest common subsequence (LCSS), multi-dimensional scaling (MDS), and density-based spatial clustering of applications with noise (DBSCAN). Our experiments, carried out with AIS data for the Lombok Strait, Indonesia, showed that the trajectory compression with DP significantly accelerates the similarity measurement process. Moreover, we found that the LCSS is the optimal algorithm for similarity measurement of vessel trajectories based on AIS data. We also applied the right combination of MDS and DBSCAN in density-based clustering. The proposed framework can distinguish trajectoriess in different directions, identify the noise, and produce good quality clusters in relatively fast total processing time.&lt;/span&gt;

An efficient and scalable density-based clustering algorithm for datasets with complex structures

PurposeRecent investigations failed to reproduce the positive rotor-guided ablation outcomes shown by initial studies for treating persistent atrial fibrillation (persAF). Phase singularity (PS) is an important feature for AF driver detection, but algorithms for automated PS identification differ. We aim to investigate the performance of four different techniques for automated PS detection.Methods2048-channel virtual electrogram (VEGM) and electrocardiogram signals were collected for 30 s from 10 patients undergoing persAF ablation. QRST-subtraction was performed and VEGMs were processed using sinusoidal wavelet reconstruction. The phase was obtained using Hilbert transform. PSs were detected using four algorithms: (1) 2D image processing based and neighbor-indexing algorithm; (2) 3D neighbor-indexing algorithm; (3) 2D kernel convolutional algorithm estimating topological charge; (4) topological charge estimation on 3D mesh. PS annotations were compared using the structural similarity index (SSIM) and Pearson’s correlation coefficient (CORR). Optimized parameters to improve detection accuracy were found for all four algorithms using Fβ score and 10-fold cross-validation compared with manual annotation. Local clustering with density-based spatial clustering of applications with noise (DBSCAN) was proposed to improve algorithms 3 and 4.ResultsThe PS density maps created by each algorithm with default parameters were poorly correlated. Phase gradient threshold and search radius (or kernels) were shown to affect PS detections. The processing times for the algorithms were significantly different (p < 0.0001). The Fβ scores for algorithms 1, 2, 3, 3 + DBSCAN, 4 and 4 + DBSCAN were 0.547, 0.645, 0.742, 0.828, 0.656, and 0.831. Algorithm 4 + DBSCAN achieved the best classification performance with acceptable processing time (2.0 ± 0.3 s).ConclusionAF driver identification is dependent on the PS detection algorithms and their parameters, which could explain some of the inconsistencies in rotor-guided ablation outcomes in different studies. For 3D triangulated meshes, algorithm 4 + DBSCAN with optimal parameters was the best solution for real-time, automated PS detection due to accuracy and speed. Similarly, algorithm 3 + DBSCAN with optimal parameters is preferred for uniform 2D meshes. Such algorithms – and parameters – should be preferred in future clinical studies for identifying AF drivers and minimizing methodological heterogeneities. This would facilitate comparisons in rotor-guided ablation outcomes in future works.

Background: This study focuses on the latest knowledge regarding Micro, Small and Medium Enterprises (MSMEs) as a current central issue. These enterprises have shown their significance in providing employment opportunities and contributing to the country’s economy. However, MSMEs face various challenges that must be addressed to optimize their outcomes. Understanding the characteristics of this group was crucial in formulating effective strategies. Objective: This study proposed to cluster or combine micro, small, and medium enterprises (MSMEs) data in a particular area based on asset value and turnover. As a result, this study aimed to gain insights into the MSME landscape in the area and provided valuable information for decision-makers and stakeholders. Methods: This study utilized two methods, namely the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method and the K-Means method. These methods were chosen for their distinct capabilities. DBSCAN was selected for its ability to handle noisy data and identify clusters with diverse forms, while K-Means was chosen for its popularity and ability to group data based on proximity. The study used a dataset containing MSME information, including asset values and turnover, collected from various sources. Results: The outcomes encompassed identifying clusters of MSMEs based on their closeness in the feature space within a specific region. Optimizing the clustering outcomes involved modifying algorithm parameters like epsilon and minimum points for DBSCAN and the number of clusters for K-Means. Furthermore, this study attained a deeper understanding of the arrangement and characteristics of MSME clusters in the region through a comparative analysis of the two methodologies. Conclusion: This study offered perspectives on clustering MSMEs based on asset value and turnover in a specific region. Employing DBSCAN and K-Means methodologies allowed researchers to depict the MSME landscape and grasp the business attributes of these enterprises. These results could aid in decision-making and strategic planning concerning the advancement of the MSME sector in the mentioned area. Future study may investigate supplementary factors and variables to deepen comprehension of MSME clusters and promote regional growth and sustainability. Keywords: Asset Value, Clustering, DBSCAN, K-Means, Turnover

Clustering samples based on similarity remains a significant challenge, especially when the goal is to accurately capture the underlying data clusters of complex arbitrary shapes. Existing density-based clustering techniques are known to be best suited for capturing arbitrarily shaped clusters. However, a key limitation of these methods is the difficulty in automatically finding the optimal set of parameters adapted to dataset characteristics, which becomes even more challenging when the data contain inherent noise. In our recent work, we proposed a Differential Evolution-based DENsity CLUstEring (DE-DENCLUE) to optimise DENCLUE parameters. This study evaluates DE-DENCLUE for its robustness in finding accurate clusters in the presence of noise in the data. DE-DENCLUE performance is compared against three other density-based clustering algorithms—DPC based on weighted local density sequence and nearest neighbour assignment (DPCSA), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), and Variable Kernel Density Estimation–based DENCLUE (VDENCLUE)—across several datasets (i.e., synthetic and real). The study has consistently shown superior results for DE-DENCLUE compared to other models for most datasets with different noise levels. Clustering quality metrics such as the Silhouette Index (SI), Davies–Bouldin Index (DBI), Adjusted Rand Index (ARI), and Adjusted Mutual Information (AMI) consistently show superior SI, ARI, and AMI values across most datasets at different noise levels. However, in some cases regarding DBI, the DPCSA performed better. In conclusion, the proposed method offers a reliable and noise-resilient clustering solution for complex datasets.

Design of computationally efficient density-based clustering algorithms

In situ mineralogical and chemical analyses of rock samples using a space-prototype laser ablation ionization mass spectrometer along with unsupervised machine learning are powerful tools for the study of surface samples on planetary bodies. This potential is demonstrated through the examination of a thin section of a terrestrial rock sample in the laboratory. Autonomous isolation of mineral phases within the acquired mass spectrometric data is achieved with two dimensionality reduction techniques: uniform manifold approximation and projection (UMAP) and density-preserving variation of UMAP (densMAP), and the density-based clustering algorithm Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN). Both densMAP and UMAP yield comparable outcomes, successfully isolating the major mineral phases fluorapatite, calcite, and forsterite in the studied rock sample. Notably, densMAP reveals additional insights into the composition of the sample through outlier detection, uncovering signals from the trace minerals pyrite, rutile, baddeleyite, and uranothorianite. Through a grid search, the stability of the methods over a broad model parameter space is confirmed, revealing a correlation between the level of data preprocessing and the resulting clustering quality. Consequently, these methods represent effective strategies for data reduction, highlighting their potential application on board spacecraft to obtain direct and quantitative information on the chemical composition and mineralogy of planetary surfaces and to optimize mission returns through the unsupervised selection of valuable data.

The density based notion for clustering approach is used widely due to its easy implementation and ability to detect arbitrary shaped clusters in the presence of noisy data points without requiring prior knowledge of the number of clusters to be identified. Density-based spatial clustering of applications with noise (DBSCAN) is the first algorithm proposed in the literature that uses density based notion for cluster detection. Since most of the real data set, today contains feature space of adjacent nested clusters, clearly DBSCAN is not suitable to detect variable adjacent density clusters due to the use of global density parameter neighborhood radius Nrad and minimum number of points in neighborhood Npts. So the efficiency of DBSCAN depends on these initial parameter settings, for DBSCAN to work properly, the neighborhood radius must be less than the distance between two clusters otherwise algorithm merges two clusters and detects them as a single cluster. Through this paper: 1) We have proposed improved version of DBSCAN algorithm to detect clusters of varying density adjacent clusters by using the concept of neighborhood difference and using the notion of density based approach without introducing much additional computational complexity to original DBSCAN algorithm. 2) We validated our experimental results using one of our authors recently proposed space density indexing (SDI) internal cluster measure to demonstrate the quality of proposed clustering method. Also our experimental results suggested that proposed method is effective in detecting variable density adjacent nested clusters.