Efficient Data Interpretation and Compression over RFID Streams

A test pattern compression scheme for test data volume and application time reduction is proposed. While compression reduces test data volume, the increased number of internal scan chains due to an on-chip, fixed-rate decompressor reduces test application time proportionately. Through on-chip decompression, both the number of virtual scan chains visible to the ATE and the functionality of the ATE are retained intact. Complete fault coverage is guaranteed by constructing the decompression hardware deterministically through analysis of the test pattern set.

3D single-photon LiDAR imaging has an important role in many applications. However, full deployment of this modality will require the analysis of low signal to noise ratio target returns and very high volume of data. This is particularly evident when imaging through obscurants or in high ambient background light conditions. This paper proposes a multiscale approach for 3D surface detection from the photon timing histogram to permit a significant reduction in data volume. The resulting surfaces are background-free and can be used to infer depth and reflectivity information about the target. We demonstrate this by proposing a hierarchical Bayesian model for 3D reconstruction and spectral classification of multispectral single-photon LiDAR data. The reconstruction method promotes spatial correlation between point-cloud estimates and uses a coordinate gradient descent algorithm for parameter estimation. Results on simulated and real data show the benefits of the proposed target detection and reconstruction approaches when compared to state-of-the-art processing algorithms.

Various compression schemes have been suggested for storage and distribution of hyperspectral remotely sensed data. Hyperspectral forestry applications that rely on the measurement of subtle variations in the spectral signature of the forest canopy can be affected by modification of the spectra induced by compression. As part of an experiment for the Canadian Space Agency (CSA), Hyperion data cubes acquired over the Greater Victoria Watershed District (GVWD) were compressed using the Successive Approximation Multi-stage Vector Quantization (SAMVQ) and Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ) algorithms developed by CSA. The data were compressed using compression ratios 10:1 and 20:1 and were returned uncompressed. The data cubes were classified into forest species using the same supervised classification methodology as applied to the original data. The classification accuracies were compared. For some applications, one can achieve significant reductions in data volume through compression. Of the compression algorithms and ratios tested, SAMVQ 10:1 has the least overall effect but still reduces classification accuracies on difficult to separate classes. While uncompressed data are preferred, SAMVQ 10:1 compression may be suitable for forest inventory.

Today’s scientific simulations generate exceptionally large volumes of data, challenging the capacities of available I/O bandwidth and storage space. This necessitates a substantial reduction in data volume, for which error-bounded lossy compression has emerged as a highly effective strategy. A crucial metric for assessing the efficacy of lossy compression is visualization. Despite extensive research on the impact of compression on visualization, there is a notable gap in the literature concerning the effects of compression on the visualization of Adaptive Mesh Refinement (AMR) data. AMR has proven to be a potent solution for addressing the rising computational intensity and the explosive growth in data volume that requires storage and transmission. However, the hierarchical and multi-resolution characteristics of AMR data introduce unique challenges to its visualization, and these challenges are further compounded when data compression comes into play. This article delves into the intricacies of how data compression influences and introduces novel challenges to the visualization of AMR data.

Somatostatin, Estrogen, and Polycystic Liver Disease

The paper considers the actual problem of improving neural network models for the classification of cardiovascular pathologies by compressing the information contained in electrocardiographic (ECG) signals. Due to the active implementation of artificial intelligence in medical diagnostics, the study focuses on creating a reduced ECG feature vector that allows for a significant reduction in training data volume without losing important diagnostic information. The research is dedicated to the development of an autoencoder model with a specialized architecture that combines convolutional and fully connected layers, attention layers, residual connections, and a symmetric structure with shared weights. This approach enables not only to compress the input multichannel signal but also to form a latent space from which the signal can be restored or used as a feature vector for classification. In the proposed autoencoder architecture, the latent representation is combined with the layer weight vector into a single reduced vector that contains information about both the shape and the structural features of the ECG signal. The model was trained using a composite loss function, which allows for balancing between signal reconstruction quality and classification accuracy. To evaluate the model, several alternative dimensionality reduction methods were used, including downsampling, transformation to Frank's orthogonal leads, calculation of a resultant vector, and principal component analysis (PCA). A number of alternative dimensionality reduction methods were used to test the model: reduction of the sampling frequency; transformation to Frank orthogonal derivations; calculation of the resulting vector; principal component analysis (PCA). All methods were tested on the PTB-XL dataset, which contains 12-lead ECG recordings with a wide range of cardiovascular pathologies. The effectiveness of the reduced feature vector was assessed using two models—a convolutional neural network (EcgNet) and a fully connected network with two hidden layers. The analysis of the results showed that, on average, classification accuracy using the reduced vector decreased by only 2% for EcgNet and increased by 3–6% for the fully connected network, indicating preservation of diagnostic information even with a high compression ratio of 25. In contrast, traditional dimensionality reduction methods such as PCA and orthogonal transformation showed a significant deterioration in classification quality (up to –16%). While downsampling to 75 Hz significantly reduces data volume, it also leads to the loss of high-frequency information critical for detecting ischemia and arrhythmias. Particular attention is given to ventricular and atrial late potentials, which are low-amplitude and high-frequency in nature. The constructed reduced feature vector preserved the informative characteristics of these classes, resulting in improved classification accuracy compared to traditional dimensionality reduction methods. Thus, the developed reduced feature vector demonstrates the ability to retain diagnostic information under significant ECG data compression, achieving a balance between computational efficiency, classification accuracy, and application versatility. The obtained results suggest that the use of the proposed reduced feature vector may serve as a promising solution for compact and efficient systems for automated ECG analysis, particularly in mobile or computing-limited systems, while also enabling faster development and testing of new neural network models for diagnostics.

This paper presents a new test data compression technique based on a compression code that uses exactly nine codewords. In spite of its simplicity, it provides significant reduction in test data volume and test application time. In addition, the decompression logic is very small and independent of the precomputed test data set. Our technique leaves many don't-care bits unchanged in the compressed test set, and these bits can be filled randomly to detect non-modeled faults. The proposed technique can be efficiently adopted for single- or multiple-scan chain designs to reduce test application time and pin requirement. Experimental results for ISCAS'89 benchmarks illustrate the flexibility and efficiency of the proposed technique.

Reducing data volume in big data: Parallel processing based data filtering techniques Professor Shikharesh Majumdar from Carleton University examines reducing data volume in big data, focusing on parallel processing based data filtering techniques. Volume, velocity, and variety are three well-known characteristics of big data. The large data volumes often introduce formidable challenges to processing such data in a timely and economical manner. Research on data filtering is underway under the leadership of Shikharesh Majumdar at the Real Time and Distributed Systems Research Centre at Carleton University. Users are often interested only in a subset of the raw data.

A distributed join is a fundamental operation for processing massive datasets in parallel. Unfortunately, computing an equi-join over such datasets is very resource-intensive, even when done in parallel. Given this cost, the equi-join operator becomes a natural candidate for optimization using approximation techniques, which allow users to trade accuracy for latency. Finding the right approximation technique for joins, however, is a challenging task. Sampling, in particular, cannot be directly used in joins; naïvely performing a join over a sample of the dataset will not preserve statistical properties of the query result.

The DEAP-3600 experiment will search for dark matter particle interactions on liquid Argon at SNOLAB, located 2 km underground in Sudbury Ontario. The DEAP-3600 detector is currently under construction and will be instrumented with 255 Hamamatsu PMT, each channel being decoupled from the PMT high voltage by a set of custom analogue signal conditioning boards. The signal conditioning boards split the PMT signals into three outputs: high gain; low gain; and into a twelve channel analogue sum (ASUM). Three different digitisers sampling at 250 MS/s, 62.5 MS/s and 50 MS/s digitise the signal conditioning outputs, the 50 MS/s sampler being a custom 6U VME Stratix IV FPGA based module generating trigger decisions based on a pulse-shape discrimination (PSD) selection. The 250 MS/s and 62.5 MS/s digitisers are commercial CAEN V1720 and V1740 modules respectively. DAQ from the digitisers to disk is handled by the Midas system with additional software event filtering provided by DEAP-3600 specific algorithms. With a total mass of 3600 kg the expected Argon β decay background is 3.6 kHz, a key feature of the DEAP-3600 TDAQ system is the reduction in data volume from 4500 MB/s to 5 MB/s, while recording all events of interest.

Advanced algorithmic solutions are necessary to process the ever-increasing amounts of mass spectrometry data that are being generated. In this study, we describe the falcon spectrum clustering tool for efficient clustering of millions of MS/MS spectra. falcon succeeds in efficiently clustering large amounts of mass spectral data using advanced techniques for fast spectrum similarity searching. First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing. Next, the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching. The nearest neighbor indexes are used to efficiently compute a sparse pairwise distance matrix without having to exhaustively perform all pairwise spectrum comparisons within the relevant precursor mass tolerance. Finally, density-based clustering is performed to group similar spectra into clusters. Several state-of-the-art spectrum clustering tools were evaluated using a large draft human proteome data set consisting of 25 million spectra, indicating that alternative tools produce clustering results with different characteristics. Notably, falcon generates larger highly pure clusters than alternative tools, leading to a larger reduction in data volume without the loss of relevant information for more efficient downstream processing. falcon is a highly efficient spectrum clustering tool, which is publicly available as an open source under the permissive BSD license at https://github.com/bittremieux/falcon.

Abstract. This paper discusses a current issue for several experimental science disciplines, which is the Big Data Problem (BDP). This research study focused on light intensity and ranging (LiDAR) datasets, which are collected for modelling spatial features found on the surface of the earth. Currently, LiDAR datasets are known to be extremely redundant for many applications. Using a formula that allows for calculating the variance of the target-induced error (so-called T-error) caused by the discretisation and quantisation of a 3D surface as a criterion for the quantitative assessment of the fidelity of a model, the use of a Q-tree-based split of the surface is proposed for cells of various sizes depending on the fidelity requirements. A LiDAR dataset representing a 1 km x 1 km terrain surface tile using approximately 12 x 106 points was used during the experiments. The initial LiDAR dataset was used to produce a digital terrain model (DTM) at a 0.5 m x 0.5 m resolution, which was used as a reference model. Subsequently, the initial LiDAR dataset was decimated at various rates, and the resulting DTMs were compared with the reference model. The Q-tree based data structure was utilised to illustrate that the Q-tree approach allows for the production of DTMs at a ‘controlled’ fidelity with a considerable reduction in data volume.

Using computer programs to simulate protein complex formation is an important approach for better comprehension of the interaction mechanisms of such molecules. The parametric representation and the Zernike polynomial method are both compact representations of molecular surfaces based on spherical harmonics, used for visualization and comparison of molecules and for protein-protein docking. They pose, however, limitations regarding surface topology and difficulty in representing arbitrary functions. In this study, the representation capacity of such methods were refined to attain high quality reproductions and applicability to arbitrary topologies. Throughout the analysis of several algorithms, the stages of surface mesh calculation and spherical mapping were identified as highly influential on the quality of the spherical harmonics parametric representation, while high sensibility to changes in the function values were identified as an influential factor for projections in 3D Zernike. A surface calculation method that generates a highly regular mesh was adopted and paired with a modern spherical mapping algorithm to yield reconstructions with low level of distortions and high quality surfaces. Similarity among surfaces from different structures in a conformational ensemble were detected to allow sharing of portions of the description among several surfaces, with corresponding reduction in data volume and processing demand and controllable influence of distortions. A new input data organization method improved the reconstruction quality of general functions in 3D Zernike, although introducing a map to restore the function. Results indicate promising application of the methods in highly detailed protein-protein docking.

This study presents a real-time, adaptive UAV system designed to enhance ecological surveys by overcoming the trade-off between wide-area coverage and high-resolution data collection. The Modular Detection and Targeting System (MDTS) integrates thermal imaging for broad detection and high-resolution RGB zoom imaging for precise species identification. Field trials demonstrated the system’s ability to detect and record both avian and mammalian species with significantly reduced redundant data and improved survey efficiency. Compared to traditional UAV methods, the MDTS achieved over 300-fold improvements in image resolution and up to a 1000-fold reduction in data volume. The system’s modular design enables rapid adaptation to diverse ecological applications, providing classification-ready data while minimizing post-processing demands. These results highlight the MDTS as a scalable, efficient tool for wildlife monitoring and environmental research, bridging the gap between detection and actionable ecological insights.

Data visualization is one of the most powerful techniques to analyze and obtain reliable results based on the displayed outputs since it allows humans to improve decision-making by visually analyzing data behavior. Nevertheless, it could be disrupted by high data amounts in the visualization, as is the case with Parallel Coordinate Plot (PCP), where data behavior and associations of volumes of data are difficult to identify. This paper aims to reduce this issue with PCP and take advantage of metaheuristics for optimization problems through a Simulated Annealing (SA) algorithm. The proposed method was developed and tested using air pollution and meteorological variables. The obtained results presented a reduction in data volume, thus helping represent the most relevant data for the final user.KeywordsMetaheuristicParallel coordinates plotFiltering