Large-scale tandem mass spectrum clustering using fast nearest neighbor searching.

An efficient and scalable density-based clustering algorithm for datasets with complex structures

A test pattern compression scheme for test data volume and application time reduction is proposed. While compression reduces test data volume, the increased number of internal scan chains due to an on-chip, fixed-rate decompressor reduces test application time proportionately. Through on-chip decompression, both the number of virtual scan chains visible to the ATE and the functionality of the ATE are retained intact. Complete fault coverage is guaranteed by constructing the decompression hardware deterministically through analysis of the test pattern set.

Subspace clustering is the task of grouping objects based on mutual similarity in subspaces of the full-dimensional space.<br/>The INSCY algorithm extends the well-known density-based clustering algorithm DBSCAN. It finds dimensionality-unbiased non-redundant subspace clusters using a tree structure to speed up the processing of subspaces. Still, finding density-based clusters in all subspaces implies an exponential search space in the number of dimensions. Thus, the running time of INSCY is still measured in hours on even small datasets of 2000 points. For larger datasets, it becomes prohibitively expensive.<br/><br/>To benefit from INSCY for real-world sized datasets, we propose a novel GPU-parallel approach that runs on standard graphics cards. To utilize the many cores of the GPU, we need new algorithmic strategies that fit the computational model of the GPU. While the GPU provides a large number of threads, traditional algorithms incur diverging threads and poor memory alignment, both of which lead to idle time and poor runtime performance. In INSCY, extracting subspace regions from the SCY-tree structure and the density-based clustering of regions itself are thus unfit for the GPU. <br/><br/>Our novel GPU-friendly algorithm GPU-INSCY computes the same subspace clustering as INSCY at dramatically reduced runtimes. <br/>To achieve this, we devise a restructured SCY-tree index-structure and associated operations for the GPU, as well as a GPU-parallel density-based subspace clustering.<br/><br/>We experimentally show that GPU-INSCY scales well with the size of the dataset and the number of dimensions, and improves the running time of INSCY by a factor of several thousand for large datasets of high dimensionality.

During the training phase of the Nearest-Neighbor (NN) algorithm, considered the most popular Instance-Based Learning (IBL) algorithm, all training instances are stored as the description of the learned concept. IBL algorithms postpone the generalization process, which usually happens during the training phase, until the classification phase starts i.e., when an unclassified data instance needs to be classified. When training sets have a high volume of instances, to store them all becomes unfeasible mainly due to storage requirements. Several IBL algorithms overcome storage related problems by implementing data volume reduction i.e., by storing only a representative subset of the training set. The investigation described in this paper focuses on four IBL algorithms that implement data reduction, which have been empirically evaluated in data sets from the UCI Repository. Their performance, considering storage reduction and classification accuracy, are presented and discussed.

Mobile networks and smart phones have become ubiquitous in our daily life. Large amount of customer related telecom data from various sources are generated every day, from which diversified behavior patterns can be revealed, including some anomalous behaviors that are vicious. It becomes increasingly important to achieve both efficient and effective customer behavior analysis based on the telecom big data. In this paper, the Multi-faceted Telecom Customer Behavior Analysis (MTCBA) framework for anomalous telecom customer behavior detection and clustering analysis is proposed. In this framework, we further design the hierarchical Locality Sensitive Hashing-Local Outlier Factor (hierarchical LSH-LOF) scheme for suspicious customer detection, and the Autoencoders with Factorization Machines (FM-AE) structure for dimension reduction to achieve more efficient clustering. Hierarchical LSH-LOF is an improved algorithm of LOF, in which we design a hierarchical LSH process that selects the approximate k nearest neighbors from coarse to fine by gradually narrowing down the scope. Experiments show its superiority over KD-tree w.r.t searching speed. FM-AE exploits Factorization Machines for learning second order feature interactions, which we prove to be useful by designing comparative experiments with five dimension reduction algorithms. With the proposed MTCBA framework, efficient and effective telecom customer behavior analysis including anomalous customer behavior detection and clustering analysis is performed on the real world telecom operating data provided by one of the major Internet service providers (ISPs) in China. Meanwhile, interpretable clustering results of six clusters are obtained to provide valuable information for the precision marketing of telecom operators, criminal combating, and social credit system construction.

Design of computationally efficient density-based clustering algorithms

Today’s scientific simulations generate exceptionally large volumes of data, challenging the capacities of available I/O bandwidth and storage space. This necessitates a substantial reduction in data volume, for which error-bounded lossy compression has emerged as a highly effective strategy. A crucial metric for assessing the efficacy of lossy compression is visualization. Despite extensive research on the impact of compression on visualization, there is a notable gap in the literature concerning the effects of compression on the visualization of Adaptive Mesh Refinement (AMR) data. AMR has proven to be a potent solution for addressing the rising computational intensity and the explosive growth in data volume that requires storage and transmission. However, the hierarchical and multi-resolution characteristics of AMR data introduce unique challenges to its visualization, and these challenges are further compounded when data compression comes into play. This article delves into the intricacies of how data compression influences and introduces novel challenges to the visualization of AMR data.

Reducing data volume in big data: Parallel processing based data filtering techniques Professor Shikharesh Majumdar from Carleton University examines reducing data volume in big data, focusing on parallel processing based data filtering techniques. Volume, velocity, and variety are three well-known characteristics of big data. The large data volumes often introduce formidable challenges to processing such data in a timely and economical manner. Research on data filtering is underway under the leadership of Shikharesh Majumdar at the Real Time and Distributed Systems Research Centre at Carleton University. Users are often interested only in a subset of the raw data.

The DEAP-3600 experiment will search for dark matter particle interactions on liquid Argon at SNOLAB, located 2 km underground in Sudbury Ontario. The DEAP-3600 detector is currently under construction and will be instrumented with 255 Hamamatsu PMT, each channel being decoupled from the PMT high voltage by a set of custom analogue signal conditioning boards. The signal conditioning boards split the PMT signals into three outputs: high gain; low gain; and into a twelve channel analogue sum (ASUM). Three different digitisers sampling at 250 MS/s, 62.5 MS/s and 50 MS/s digitise the signal conditioning outputs, the 50 MS/s sampler being a custom 6U VME Stratix IV FPGA based module generating trigger decisions based on a pulse-shape discrimination (PSD) selection. The 250 MS/s and 62.5 MS/s digitisers are commercial CAEN V1720 and V1740 modules respectively. DAQ from the digitisers to disk is handled by the Midas system with additional software event filtering provided by DEAP-3600 specific algorithms. With a total mass of 3600 kg the expected Argon β decay background is 3.6 kHz, a key feature of the DEAP-3600 TDAQ system is the reduction in data volume from 4500 MB/s to 5 MB/s, while recording all events of interest.

3D single-photon LiDAR imaging has an important role in many applications. However, full deployment of this modality will require the analysis of low signal to noise ratio target returns and very high volume of data. This is particularly evident when imaging through obscurants or in high ambient background light conditions. This paper proposes a multiscale approach for 3D surface detection from the photon timing histogram to permit a significant reduction in data volume. The resulting surfaces are background-free and can be used to infer depth and reflectivity information about the target. We demonstrate this by proposing a hierarchical Bayesian model for 3D reconstruction and spectral classification of multispectral single-photon LiDAR data. The reconstruction method promotes spatial correlation between point-cloud estimates and uses a coordinate gradient descent algorithm for parameter estimation. Results on simulated and real data show the benefits of the proposed target detection and reconstruction approaches when compared to state-of-the-art processing algorithms.

Despite its promise, RFID technology presents numerous challenges, including incomplete data, lack of location and containment information, and very high volumes. In this work, we present a novel data interpretation and compression substrate over RFID streams to address these challenges in enterprise supply-chain environments. Our results show that our inference techniques provide good accuracy while retaining efficiency, and our compression algorithm yields significant reduction in data volume.

Various compression schemes have been suggested for storage and distribution of hyperspectral remotely sensed data. Hyperspectral forestry applications that rely on the measurement of subtle variations in the spectral signature of the forest canopy can be affected by modification of the spectra induced by compression. As part of an experiment for the Canadian Space Agency (CSA), Hyperion data cubes acquired over the Greater Victoria Watershed District (GVWD) were compressed using the Successive Approximation Multi-stage Vector Quantization (SAMVQ) and Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ) algorithms developed by CSA. The data were compressed using compression ratios 10:1 and 20:1 and were returned uncompressed. The data cubes were classified into forest species using the same supervised classification methodology as applied to the original data. The classification accuracies were compared. For some applications, one can achieve significant reductions in data volume through compression. Of the compression algorithms and ratios tested, SAMVQ 10:1 has the least overall effect but still reduces classification accuracies on difficult to separate classes. While uncompressed data are preferred, SAMVQ 10:1 compression may be suitable for forest inventory.

Abstract. This paper discusses a current issue for several experimental science disciplines, which is the Big Data Problem (BDP). This research study focused on light intensity and ranging (LiDAR) datasets, which are collected for modelling spatial features found on the surface of the earth. Currently, LiDAR datasets are known to be extremely redundant for many applications. Using a formula that allows for calculating the variance of the target-induced error (so-called T-error) caused by the discretisation and quantisation of a 3D surface as a criterion for the quantitative assessment of the fidelity of a model, the use of a Q-tree-based split of the surface is proposed for cells of various sizes depending on the fidelity requirements. A LiDAR dataset representing a 1 km x 1 km terrain surface tile using approximately 12 x 106 points was used during the experiments. The initial LiDAR dataset was used to produce a digital terrain model (DTM) at a 0.5 m x 0.5 m resolution, which was used as a reference model. Subsequently, the initial LiDAR dataset was decimated at various rates, and the resulting DTMs were compared with the reference model. The Q-tree based data structure was utilised to illustrate that the Q-tree approach allows for the production of DTMs at a ‘controlled’ fidelity with a considerable reduction in data volume.

Using computer programs to simulate protein complex formation is an important approach for better comprehension of the interaction mechanisms of such molecules. The parametric representation and the Zernike polynomial method are both compact representations of molecular surfaces based on spherical harmonics, used for visualization and comparison of molecules and for protein-protein docking. They pose, however, limitations regarding surface topology and difficulty in representing arbitrary functions. In this study, the representation capacity of such methods were refined to attain high quality reproductions and applicability to arbitrary topologies. Throughout the analysis of several algorithms, the stages of surface mesh calculation and spherical mapping were identified as highly influential on the quality of the spherical harmonics parametric representation, while high sensibility to changes in the function values were identified as an influential factor for projections in 3D Zernike. A surface calculation method that generates a highly regular mesh was adopted and paired with a modern spherical mapping algorithm to yield reconstructions with low level of distortions and high quality surfaces. Similarity among surfaces from different structures in a conformational ensemble were detected to allow sharing of portions of the description among several surfaces, with corresponding reduction in data volume and processing demand and controllable influence of distortions. A new input data organization method improved the reconstruction quality of general functions in 3D Zernike, although introducing a map to restore the function. Results indicate promising application of the methods in highly detailed protein-protein docking.

This study presents a real-time, adaptive UAV system designed to enhance ecological surveys by overcoming the trade-off between wide-area coverage and high-resolution data collection. The Modular Detection and Targeting System (MDTS) integrates thermal imaging for broad detection and high-resolution RGB zoom imaging for precise species identification. Field trials demonstrated the system’s ability to detect and record both avian and mammalian species with significantly reduced redundant data and improved survey efficiency. Compared to traditional UAV methods, the MDTS achieved over 300-fold improvements in image resolution and up to a 1000-fold reduction in data volume. The system’s modular design enables rapid adaptation to diverse ecological applications, providing classification-ready data while minimizing post-processing demands. These results highlight the MDTS as a scalable, efficient tool for wildlife monitoring and environmental research, bridging the gap between detection and actionable ecological insights.