Forensics Aware Lossless Compression of CAN Traffic Logs

Time series compression encodes the information in a time-ordered sequence of data points into fewer bits, thereby reducing storage costs and possibly other costs. Compression methods are either general or XOR-based. General compression methods are time-consuming and are not suitable in streaming scenarios, while XOR-based methods are unable to consistently maintain high compression ratios. Further, existing methods compress the integer and decimal parts of floating-point values as a whole, thus disregarding the different characteristics of the two parts. We propose Camel , a new compression method for floating-point time series with the goal of advancing the compression ratios and efficiency achievable. Camel compresses the integer and decimal parts of the double-precision floating-point numbers in time series separately; and instead of performing XOR operations on values using their previous value, Camel identifies values that enable higher compression ratios. Camel also includes means of indexing compressed data, thereby making it possible to query compressed data efficiently. We report on an empirical study of Camel and 11 lossless and 6 lossy compression methods on 22 public datasets and three industrial datasets from AliCloud. The study offers evidence that Camel is capable of outperforming existing methods in terms of both compression ratio and efficiency and is capable of excellent compression performance on both time series and non-time series data.

A hybrid compression method for the NMR data based on window averaging and Discrete Cosine Transform

We use both lossy and lossless compression methods to reduce large amounts of digital data, making it easier to store and manage. Lossy and lossless compression is frequently used in astronomy, to condense the large amount of data collected by instruments namely, the James Webb Space Telescope and the Hubble Space Telescope. Similarly to diffraction images, astronomical images have large areas of empty space since stellar objects are very bright, yet show very small in images, which after a while, takes up a needless amount of limited digital storage space. Lossy compression achieves higher compression ratios but sacrifices some data quality in the process, while lossless compression preserves the data’s original form, but results in less storage savings. This work, conducted at Brookhaven National Laboratory’s National Synchrotron Light Source-II at the AMX and FMX macromolecular beam lines, aimed to evaluate the effectiveness of the lossy compression method known as Wavelet Compression. Wavelet compression is a method of data compression that uses wavelet transforms to reduce the size of data while preserving its essential features. Crystals are repetitive molecular assemblies that create diffraction patterns when exposed to X-rays, while detectors capture the amplitude of diffraction pattern spots (reflections). Similar to astronomical images, diffraction images contain large areas of empty space. This unnecessary data can quickly consume limited digital storage. By studying and optimizing compression methods, we not only improve data management but also advancements in various scientific disciplines.

Summary form only given. We propose a novel compression method, static PPM (SP), which supports partial decode (p-decode) with low memory and computation requirement. P-decode is a function that decodes data from an arbitrary position without decoding the whole, which is critical for exploiting large data in a compressed state. While conventional compression methods do not support p-decode, recent self-indexing data structures, such as compressed suffix arrays (CSA) (Sadakane, K., J. Algorithms, vol.48, no.2, p.294-313, 2003) and FM-index (Ferragina, P. and Manzini, G. Proc. ACM-SIAM SODA, p.269-78, 2001), support p-decode. However, they have to store the whole compressed data in memory, since the order of the data is not preserved after compression. This causes a serious problem when the compressed data is larger than memory size. In contrast, SP does not have to store compressed data in memory and, thus, is the first compression method that supports p-decode for very large compressed data. In order to show SP's high compression performance inherited from PPM and its p-decode performance, we compared SP, CSA, FM-index, gzip (with option -9) and bzip (with option -9) in terms of speed and compression ratio. The results show that SP achieves fast p-decode with high compression ratio.

INTRODUCTION: Genetic data plays a key role in the healthcare area in specific, but they are typically very large in size. Many research shows that absence of DNA information at the right time is one of the major causes of error in the healthcare area. The more genomics information that analysts secure, the better the prospects for individual and general wellbeing. Persevering and retrieving genetic information in the right form within the given time is a big challenge in the field of Healthcare. Effectively, pre-birth DNA tests screen for formative variations from the norm. Before long, patients will have their blood sequenced to detect any nonhuman DNA that may flag an irresistible illness. Later on, somebody managing malignancy will most likely track the movement of the sickness by having the DNA and RNA of single cells from various tissues sequenced every day. DNA sequencing of whole population will give a complete and better prediction of population wellbeing.OBJECTIVES: Hereditary data is growing exponentially; hence it is hard to deal with the consistently developing hereditary database. The human genome in its base configuration occupies almost thirty terabyte of storage space. Computational assets are constrained. Not just storage, transmission abilities and run time memory is likewise constrained. Data Compression is a test when the hereditary information is exponentially expanding. It is critical to save the integrity of hereditary information while packing it. Hence the main objective of this paper is to develop a lossless DNA compression algorithm that not only gives better compression but also help in retrieval of Information for efficient use in the area of Healthcare.METHODS: In this paper a lossless hereditary data compression method is being proposed. The proposed calculation works in a horizontal mode and utilization a reference based substitution technique for compression. The principle thought of this paper is in the kind of similarity scanned. All the predominant hereditary Compression methods search for similarity within the chromosome. These calculations either pursue flat mode or vertical mode for accomplishing compression. But whichever method the existing genetic compression algorithms use, they are all based on searching similarities within the chromosome i.e. they exploit only inter chromosomal similarities. The current studies focus will show that compression ratio achieved by analyzing individual chromosome is always less than the method in which we analyze and compress intra chromosomal similarities.RESULTS: This study shows that by simply using exactly matching repeats amongst all the chromosomes of the same genome, not only the compression ratio is improving but also a detailed study of all the similarities and differences between two genomes of the same species can be conducted.CONCLUSION: In this study, a new compression algorithm is being proposed for compressing DNA. Along with Inter chromosomal similarities, Intra chromosomal similarities are considered for this method. The results clearly shows that intra chromosomal matches are bigger and more than inter chromosomal matches which helps us to achieve better compression ratio.

When detecting a target, the radar receives a large amount of complex data. Through three-dimensional imaging, the large amount of data can be converted into relevant features of the scattering center for storage, thereby realizing data compression. This paper proposes a radar data compression method based on scattering center extraction. In this paper, a mathematical model is established, and the scattering center is extracted by the three-dimensional ESPRIT imaging method, and the number of scattering centers is estimated. A sweep frequency reconstruction algorithm is used on the compressed data to verify the recoverability of the compressed data. By using the method of this paper, a higher compression ratio can be obtained while ensuring smaller data errors. Experimental data processing results validate the conclusions of this paper.

The traffic of mobile network environment is increasing day by day. When a large amount of data need to be transmitted, the low-speed wireless network today is hard to meet users' needs. Long response time of operation and high requirements of terminal's performance will keep user annoy and away. Existing independent data compression and transmission algorithm can't solve this problem effectively. In this paper, data compression and transmission considered as a whole, a new kind of data compression and transmission method was designed in this paper to solve the problem and achieved good results finally.

Ion mobility spectrometry is a rapid scanning measurement method for which compression methods that facilitate the handling of large collections of data are beneficial. Peak distortion in reconstructed ion mobility spectra from linear wavelet compression is problematic in that artifact peaks may cause false positive alarms. Peak shifting also may cause false alarms if target peaks shift out of or interfering peaks shift into detection windows. Nonlinear wavelet compression (NLWC) preserves peak shape and can lessen the degree of distortion, shifting, and artifact peaks in the reconstructed spectra. NLWC was applied to achieve high compression and fidelity in the reconstructed spectra. Another benefit is that NLWC improves signal-to-noise ratios and thus the models built from compressed data are improved. By compressing both the drift time order and the spectrum acquisition order, greater compressions maybe achieved. A two-way nonlinear wavelet compression method that incorporates alternating least squares (2W-NLWC-ALS) algorithm was devised by applying ALS to partially reconstructed wavelet coefficients generated from two-way NLWC. The number of components in a data set can be determined automatically using ASIMPLISMA. The smaller ALS models are saved as the final compressed data and can be used to reconstruct the entire data set efficiently without maintaining the compressed wavelet coefficient matrix of the original data set. The 2W-NLWC-ALS algorithm provides greater compression ratios compared to regular wavelet compression and interpretable models. Using this method, large volumes of data can be acquired and easily evaluated through a simple compressed model. A compression ratio of 510 ppm, root-mean-square error (E(RMS)) of 6.3 mV (full-scale signal is usually 1 V or larger), and relative root-mean-square error (RE(RMS)) of 1.62% were achieved for data sets collected by CAM. A compression ratio of 46 ppm, E(RMS) of 9.2 mV, and RE(RMS) of 0.42% were achieved for data sets collected with an ITEMISER instrument. The 2W-NLWC-ALS algorithm is an efficient compression method that provides the benefits of a simple model.

The increasing test data volume is considered as a biggest challenge in circuit under test. This challenge leads to higher computational complexity associated with the testing circuits. To reduce the testing time and addresses the problem related to high overhead, high fault coverage andincreased power dissipationmethod is used in system-on-chip (SoC). A new test compression method called divide and compress (D&C) lossless compression coder with Discrete Cosine Transformis used to enhance the compression capability. This framework of D&C lossless compression coder combines the lossless and lossy compression methods. The Discrete Cosine Transform (DCT) manipulates each data bit inside file and this reduces the file size without any data loss after decoding to lossless compression.Depending on the data characteristic, the compression of data is done without any loss in data. Further, the decompression is attained using advanced Finite State Machine (FSM) to reduce overhead. The experimental results on large benchmark circuits like ITC’99 and ISCAS’99 proves that the proposed D&C lossless compression attains reduced testing time and better compression ratio. Further, itlays only soft burden on the hardware.

This paper proposes a method for avoiding large amount of data saving and reading during the reconstruction of forward-backward time-stepping (FBTS) method using data compression and decompression method. Data compression and decompression are conducted by the cosine transform. A reconstruction example is shown to demonstrate the effect of the proposed method. The reconstruction result with data compression and decompression is acceptable if the compression ratio is less than 90%.

Data compression techniques in IoT-enabled wireless body sensor networks: A systematic literature review and research trends for QoS improvement

In this paper, a novel ECG data compression method is presented which employs set partitioning in hierarchical trees algorithm (SPIHT), sub-band energy compression (SEC) method and two-dimensional electrocardiogram (2D-ECG). The 2D-ECG is a two-dimensioned array, which each row of this array indicates one or more period and amplitude normalized (PANed) ECG beats. In the previous works, SPIHT and sub-band energy compression method have used to compress one or two-dimensional signals separately and have shown their efficiencies such as precise rate control, progressive quality, high compression ratio and low root mean square difference (PRD). In here, we put these two methods together and illustrate that they can be applied to 2D-ECG to achieve better results and this is because of the wavelet transform eliminating effect on redundancies between adjacent samples - it is also eliminated in one-dimensional ECG - and between adjacent beats by applying 2D wavelet transform.

<p>The projection into 2030 of the climate data volume increase brings an important challenge to the climate science community. This is particularly true for the CMIP7 that is projected to need about an Exabyte of storage capacity. Error-bounded lossy compression is explored as a potential solution to the above problem by different climate research teams. Several lossy compression schemes have been proposed leveraging different forms of decorrelation (transforms, prediction, HoSVD, DNN), quantization (linear, non-linear, vector), and encoding (dictionary-based, variable length, etc.) algorithms. Our experience with different applications shows that the compression methods often need to be customized and optimized to fit the specificities of the datasets to compress and the user requirements on the compression quality, ratio, and throughput. However, none of the existing lossy compression software for scientific data has been designed to be customizable. To address this issue, we developed SZ3, an innovative customizable, modular compression framework. SZ3 is a full C++ refactoring of SZ2 enabling the specialization, addition, or removal of each stage of the lossy compression pipeline to fit the specific characteristics of the datasets to compress and the use-case requirements. This extreme flexibility allows adapting SZ3 to many different use-cases, from ultra-high compression for visualization to ultra-high-speed compression between the CPU (or GPU) and the memory. Thanks to its unique set of features: customization, high compression ratio, high compression throughput, and excellent accuracy preservation, SZ3 won a 2021 R&D100 award. In this presentation, we present SZ3 and a new data prediction-based decorrelation method that significantly improves the compression ratios for climate datasets over the state-of-the-art lossy compressors, while preserving the same data accuracy. Experiments based on CESM datasets show that SZ3 can lead to up to 300% higher compression ratios than SZ2 with the same compression error bound and similar compression throughput.</p>

Image compression is the application of data compression techniques to two-dimensional digital images I(x, y) , to reduce the redundancy of the image data for storage or transmission in an efficient form. Image compression can be classified into two categories: lossless or lossy. Lossless compression, which achieves smaller compression ratios than lossy compression, mainly takes advantage of the image contents containing a non uniform probability distribution for a variable-length representation of the image pixels. Such images include technical drawings, icons or comics, and high-value contents such as medical imagery or image scans made for archival purposes. However, lossy compression methods, especially when they achieve a very high compression ratio, can introduce compression artifacts. Nevertheless, lossy compressions are especially suitable for natural images, such as photos, in applications where a minor (sometimes imperceptible) loss of fidelity is acceptable when it is desirable to achieve a substantial reduction in bitrate. Most of the stateof-the-art image compression standards use a combination of lossy and lossless algorithms to achieve the best performance. The Joint Photographic Experts Group (JPEG) [1], a discrete cosine transform(DCT)-based technique, is the most widely used standardized lossy image compression mechanism; it was designed for compressing either full-color or gray-scale images of natural, real-world, scenes. It works well for photographs, naturalistic artwork and similar material if the compression ratio is about 20 : 1, which is much better than the 4 : 1 compression ratio provided by a lossless compression method such as the Graphics Interchange Format (GIF) [2]. However, it does not work so well for lettering, simple cartoons, or line drawings.

This paper conducts a comparative analysis of the impact of unsparsing by data scaling on lossless data compression methods. The most commonly used data compression algorithms such as gzip, zlib, bzip2, and lzma are tested by their compression efficiency in unsparsed datasets by the performance metrics such as compression time, decompression time, and compression ratio for both the original and scaled datasets. Five different data scaling techniques are investigated to transform the original data into unsparsed data such as min-max, robust, absolute max, standardization, and normalization. Our research reveals relationships between data scaling methodologies and compression performance, with disparities in compression efficiency and computational complexity. Fur-thermore, we investigate the effects of scaling on compression ratio and provide results for improving the understanding of the factors that influence lossless data compression methods for non-sparse tabular data.