Hierarchical Residual Encoding for Multiresolution Time Series Compression

We present our initial findings regarding the problem of the impact that time series compression may have on similarity-queries, in the settings in which the elements of the dataset are accompanied with additional contexts. Broadly, the main objective of any data compression approach is to provide a more compact (i.e., smaller size) representation of a given original dataset. However, as has been observed in the large body of works on compression of spatial data, applying a particular algorithm “blindly” may yield outcomes that defy the intuitive expectations – e.g., distorting certain topological relationships that exist in the “raw” data [7]. In this study, we quantify this distortion by defining a measure of similarity distortion based on Kendall’s \(\tau \). We evaluate this measure, and the correspondingly achieved compression ratio for the five most commonly used time series compression algorithms and the three most common time series similarity measures. We report some of our observations here, along with the discussion of the possible broader impacts and the challenges that we plan to address in the future.

Time series compression encodes the information in a time-ordered sequence of data points into fewer bits, thereby reducing storage costs and possibly other costs. Compression methods are either general or XOR-based. General compression methods are time-consuming and are not suitable in streaming scenarios, while XOR-based methods are unable to consistently maintain high compression ratios. Further, existing methods compress the integer and decimal parts of floating-point values as a whole, thus disregarding the different characteristics of the two parts. We propose Camel , a new compression method for floating-point time series with the goal of advancing the compression ratios and efficiency achievable. Camel compresses the integer and decimal parts of the double-precision floating-point numbers in time series separately; and instead of performing XOR operations on values using their previous value, Camel identifies values that enable higher compression ratios. Camel also includes means of indexing compressed data, thereby making it possible to query compressed data efficiently. We report on an empirical study of Camel and 11 lossless and 6 lossy compression methods on 22 public datasets and three industrial datasets from AliCloud. The study offers evidence that Camel is capable of outperforming existing methods in terms of both compression ratio and efficiency and is capable of excellent compression performance on both time series and non-time series data.

With the advances in information science, vast amounts of financial time series data can been collected and analyzed. In modern time series analysis, sequential pattern mining (SPM) and association discovery (AD) are the most important techniques to predict the future trends. This study aims at developing advanced SPM and AD for financial data by cutting edge techniques from artificial intelligence and machine learning. The nonlinearity and non-stationarity of financial time series dynamics pose a major challenge for SPM and AD. This study employs time–frequency analysis to extract features for SPM. Then, a sparse multi-manifold clustering (SMMC) is used to partition the feature space into several disjointed regions for better AD. Finally, local relevance vector machines (RVMs) are employed for AD and perform the forecasting. Different from traditional methods, the novel forecasting system operates on multiple resolutions and multiple dynamic regimes. SMMC finds both the neighbors and the weights automatically by a sparse solution, which approximately spans a low-dimensional affine subspace at that point. RVM, the Bayesian kernel machines, can produce parsimonious models with excellent generalization properties. Taking multiple time series data from financial markets as an example, the empirical results demonstrate that the proposed model outperforms traditional models and significantly reduces the forecasting errors. The framework is effective and suitable for other time series forecasting.

Chapter 1 - Introduction

Digital image have been expanding in usage. The importance of image compression algorithms have been increased, to produce smaller file size with a lower computational requirement. The wavelet transform offers a natural decomposition of images at multiple resolutions, since the wavelet transform can decompose images into various multiresolution subbands, where the correlation exists. A novel technique for image compression by taking advantage of correlation is addressed. It is based on predictive edge detection from the LL band of the lowest resolution level to predict the edge in the LH, HL and HH bands in the higher resolution level. If the coefficient is predictive as an edge it is preserved, otherwise it is discarded. In the decoder, the location of the preserved coefficients can also be found as in the encoder. Results showed that photo images give a higher degree of compression compared with the satellite images, due to the information in the satellite images is higher than in the photo images.

The rapid growth of digital libraries, e-governance, and internet based applications has caused an exponential escalation in the volume of ‘Big-data’ particularly due to texts, images, audios and videos that are being both archived and transmitted on a daily basis. In order to make their storage and transfer efficient, different data compression techniques are used in the literature. The ultimate motive behind data compression is to transform a big size data into small size data, which eventually implies less space while archiving, and less time in transferring. However, in order to operate/analyze compressed data, it is usually necessary to decompress it, so as to bring back the data to its original form, which unfortunately warrants an additional computing cost. In this backdrop, if operating upon the compressed data itself can be made possible without going through the stage of decompression, then the advantage that could be accomplished due to compression would escalate. Further due to compression, from the data structure and storage perspectives, the original visibility structure of the data also being lost, it turns into a potential challenge to trace the original information in the compressed representation. This challenge is the motivation behind exploring the idea of direct processing on the compressed data itself in the literature. The proposed survey paper specifically focuses on compressed document images and brings out two original contributions. The first contribution is that it presents a critical study on different image analysis and image compression techniques, and highlights the motivational reasons for pursuing document image analysis in the compressed domain. The second contribution is that it summarizes the different compressed domain techniques in the literature so far based on the type of compression and operations performed by them. Overall, the paper aims to provide a perspective for pursuing further research in the area of document image analysis and pattern recognition directly based on the compressed data.

In the scientific domain, extremely large amounts of data are generated by large-scale high performance computing (HPC) simulations. Storing and sending such vast volumes of data poses serious scalability and performance issues, which can be considerably mitigated by data compression techniques which significantly reduced storage size and data movement burdens. Since scientific data are being shared by scientists more and more frequently, data security methods that ensure the confidentiality, integrity, and availability of data itself are becoming increasingly important. As such, combing compression and encryption is critical to storing large-scale datasets securely. In this work, we explore how to integrate data compression and cryptography techniques as efficiently as possible for big scientific datasets in the HPC field. We perform thorough experiments using different scientific datasets with the state-of-the-art error-bounded lossy compressor - SZ - on a real-world supercomputing environment. Experiments verify that performing encryption before lossy compression (a.k.a., encr-cmpr method) may invalidate the advantage of compression algorithms. By contrast, executing encryption after lossy compression (a.k.a., cmpr-encr method) keeps not only high compression ratios but high overall execution speed. Experiments also verify that the encryption overhead under the cmpr-encr method decreases with increasing compression ratios, which means very good scalability.

The cost of I/O is a significant challenge on current supercomputers, and the trend is likely to continue into the foreseeable future. This challenge is amplified in scientific visualization because of the requirement to consume large amounts of data before processing can begin. Lossy compression has become an important technique in reducing the cost of performing I/O. In this paper we consider the implications of using compressed data for visualization within a scientific workflow. We use visualization operations on simulation data that is reduced using three different state-of-the-art compression techniques. We study the storage efficiency and preservation of visualization features on the resulting compressed data, and draw comparisons between the three techniques used. Our contributions can help inform both scientists and researchers in the use and design of compression techniques for preservation of important visualization details.

The volume of radio-astronomical data is a considerable burden in the processing and storing of radio observations with high time and frequency resolutions and large bandwidths. Lossy compression of interferometric radio-astronomical data is considered to reduce the volume of visibility data and to speed up processing. A new compression technique named "Dysco" is introduced that consists of two steps: a normalization step, in which grouped visibilities are normalized to have a similar distribution; and a quantization and encoding step, which rounds values to a given quantization scheme using a dithering scheme. Several non-linear quantization schemes are tested and combined with different methods for normalizing the data. Four data sets with observations from the LOFAR and MWA telescopes are processed with different processing strategies and different combinations of normalization and quantization. The effects of compression are measured in image plane. The noise added by the lossy compression technique acts like normal system noise. The accuracy of Dysco is depending on the signal-to-noise ratio of the data: noisy data can be compressed with a smaller loss of image quality. Data with typical correlator time and frequency resolutions can be compressed by a factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added system noise. An implementation of the compression technique is released that provides a Casacore storage manager and allows transparent encoding and decoding. Encoding and decoding is faster than the read/write speed of typical disks. The technique can be used for LOFAR and MWA to reduce the archival space requirements for storing observed data. Data from SKA-low will likely be compressible by the same amount as LOFAR. The same technique can be used to compress data from other telescopes, but a different bit-rate might be required.

Hyperspectral image compression has become increasingly important in data exploitation because of enormous data volumes and high redundancy provided by hundreds of contiguous spectral channels. Since a hyperspectral image can be viewed as a 3-dimensional (3D) image cube, many efforts have been devoted to extending 2D image compression techniques to perform 3D image compression on hyperspectral image cubes. Unfortunately, some major issues generally encountered in hyperspectral data exploitation at low or very low-bit rate compression, for example, subpixels and mixed pixels which do not occur in traditional pure pixel-based image compression are often overlooked in such a 2D-to-3D compression. Accordingly, a direct application of 2D-to-3D compression techniques to hyperspectral image cubes without taking precaution may result in significant loss of crucial spectral information provided by subtle substances such as small objects, anomalies during low bit-rate lossy compression. This paper takes a rather different view by investigating lossy hyperspectral compression from a perspective of exploring spectral information, referred to as exploitation-based lossy compression and further develops spectral/spatial hyperspectral image compression to effectively preserve crucial and vital spectral information of objects which are generally missed by commonly used mean-squared error (MSE) or signal-to-noise ratio (SNR)-based compression techniques when lossy compression is performed at low bit rates. In order to demonstrate advantages of the proposed spectral/spatial compression approach applications of subpixel target detection and mixed pixel analysis are used for experiments for performance evaluation.

Applicative fields based on the analysis of large images must deal with two important problems. First, the size in memory of such images usually forbids a global image analysis hereby inducing numerous problems for the design of a global image partition. Second, due to the high resolution of such images, global features only appear at low resolutions and a single resolution analysis may loose important information. The tiled top-down pyramidal model has been designed to solve this two major challenges. This model provides a hierarchical encoding of the image at single or multiple resolutions using a top-down construction scheme. Moreover, the use of tiles bounds the amount of memory required by the model while allowing global image analysis. The main limitation of this model is the splitting step used to build one additional partition from the above level. Indeed, this step requires to temporary refine the split region up to the pixel level which entails high memory requirements and processing time. In this paper, we propose a new splitting step within the tiled top-down pyramidal framework which overcomes the previously mentioned limitations.

Several cybersecurity domains, such as ransomware detection, forensics and data analysis, require methods to reliably identify encrypted data fragments. Typically, current approaches employ statistics derived from byte-level distribution, such as entropy estimation, to identify encrypted fragments. However, modern content types use compression techniques which alter data distribution pushing it closer to the uniform distribution. The result is that current approaches exhibit unreliable encryption detection performance when compressed data appear in the dataset. Furthermore, proposed approaches are typically evaluated over few data types and fragment sizes, making it hard to assess their practical applicability. This paper compares existing statistical tests on a large, standardized dataset and shows that current approaches consistently fail to distinguish encrypted and compressed data on both small and large fragment sizes. We address these shortcomings and design EnCoD, a learning-based classifier which can reliably distinguish compressed and encrypted data. We evaluate EnCoD on a dataset of 16 different file types and fragment sizes ranging from 512B to 8KB. Our results highlight that EnCoD outperforms current approaches by a wide margin, with accuracy ranging from sim 82% for 512B fragments up to sim 92% for 8KB data fragments. Moreover, EnCoD can pinpoint the exact format of a given data fragment, rather than performing only binary classification like previous approaches.

We use both lossy and lossless compression methods to reduce large amounts of digital data, making it easier to store and manage. Lossy and lossless compression is frequently used in astronomy, to condense the large amount of data collected by instruments namely, the James Webb Space Telescope and the Hubble Space Telescope. Similarly to diffraction images, astronomical images have large areas of empty space since stellar objects are very bright, yet show very small in images, which after a while, takes up a needless amount of limited digital storage space. Lossy compression achieves higher compression ratios but sacrifices some data quality in the process, while lossless compression preserves the data’s original form, but results in less storage savings. This work, conducted at Brookhaven National Laboratory’s National Synchrotron Light Source-II at the AMX and FMX macromolecular beam lines, aimed to evaluate the effectiveness of the lossy compression method known as Wavelet Compression. Wavelet compression is a method of data compression that uses wavelet transforms to reduce the size of data while preserving its essential features. Crystals are repetitive molecular assemblies that create diffraction patterns when exposed to X-rays, while detectors capture the amplitude of diffraction pattern spots (reflections). Similar to astronomical images, diffraction images contain large areas of empty space. This unnecessary data can quickly consume limited digital storage. By studying and optimizing compression methods, we not only improve data management but also advancements in various scientific disciplines.

Time series forecasting has witnessed rapid methodological evolution due to the increasing data availability in several domains such as energy, healthcare, transportation, and finance. Despite advances in forecasting models, including statistical approaches, deep neural networks, and foundation models, benchmarking remains restricted by the limited variety in datasets, the irregular structural complexity, and the lack of unified evaluation frameworks. This research addresses these limitations by presenting a large-scale, unified, and heterogeneous time series data archive that spans 12 domains, encompasses up to 2 billion time points, and offers multiple temporal resolutions. This research aims to design and implement a unified testbed that incorporates more than 100 datasets, balancing both univariate and multivariate series, and quantifies explicitly structural components such as trend, seasonality, stationarity, shifting, and transition. It goes beyond reflecting the plain scale and structural diversity of real-world time series, enabling a fair and reproducible evaluation framework of forecasting models. By providing both a full-scale and a computationally efficient down-sampled version, this work lays the foundations of a new era of benchmarking in time series forecasting. It offers in the research community a scalable and extensible infrastructure, designed to accelerate the development of robust, adaptable and generalizable prediction models that are aligned with real-world complexities.

Airborne light detection and ranging (LIDAR) system uses in many field such as military, agriculture, geology and transportation. LIDAR data acquisition consumes several gigabytes from the storage capacity of the system. So, it's necessary to reduce the on board storage capacity as well as its transmission rate. Compression of LIDAR data means reducing the amount of data required to represent the digital data acquired by LIDAR system. This is done by removing data redundancies to achieve compression. In this paper we explain LIDAR system theory and component, compare between lossy and lossless compression techniques then we discuss LIDAR data compression techniques in order to choose a suitable one for LIDAR system that gives high compression ratio with suitable processing time. In most LIDAR data compression techniques classification of LIDAR data is an important issue as a preprocessing step to simplify the data. Using more than one compression algorithms increase the compression ratio but increase the processing time as well.