A Comparative Analysis of the Lossless Data Compression Methods for Unsparsed Tabular Data

<p>The need of data compression at smart Edge/Fog-based gateways is undeniable as data compression can significantly reduce the amount of data that has to be transmitted over a network. This, in turn, has a direct impact on reducing transmission latency and increasing network bandwidth. In time-critical and data-sensitive IoT applications such as healthcare, lossless data compression is preferable as compressed data can be recovered without losing any information. However, it is not an easy task to choose a proper lossless data compression algorithm for IoT applications as each lossless data compression method has its own advantages and disadvantages. This paper focuses on the analysis of lossless data compression algorithms run at smart Edge/Fog gateways. Widely used lossless data compression is run at different hardware which is often used as smart Fog/Edge gateways. The latency of data compression and the compression rate in different cases of input data sizes are analyzed. The paper provides guidelines for choosing a proper lossless data compression algorithm for time-critical IoT applications. <br /></p>

INTRODUCTION: Genetic data plays a key role in the healthcare area in specific, but they are typically very large in size. Many research shows that absence of DNA information at the right time is one of the major causes of error in the healthcare area. The more genomics information that analysts secure, the better the prospects for individual and general wellbeing. Persevering and retrieving genetic information in the right form within the given time is a big challenge in the field of Healthcare. Effectively, pre-birth DNA tests screen for formative variations from the norm. Before long, patients will have their blood sequenced to detect any nonhuman DNA that may flag an irresistible illness. Later on, somebody managing malignancy will most likely track the movement of the sickness by having the DNA and RNA of single cells from various tissues sequenced every day. DNA sequencing of whole population will give a complete and better prediction of population wellbeing.OBJECTIVES: Hereditary data is growing exponentially; hence it is hard to deal with the consistently developing hereditary database. The human genome in its base configuration occupies almost thirty terabyte of storage space. Computational assets are constrained. Not just storage, transmission abilities and run time memory is likewise constrained. Data Compression is a test when the hereditary information is exponentially expanding. It is critical to save the integrity of hereditary information while packing it. Hence the main objective of this paper is to develop a lossless DNA compression algorithm that not only gives better compression but also help in retrieval of Information for efficient use in the area of Healthcare.METHODS: In this paper a lossless hereditary data compression method is being proposed. The proposed calculation works in a horizontal mode and utilization a reference based substitution technique for compression. The principle thought of this paper is in the kind of similarity scanned. All the predominant hereditary Compression methods search for similarity within the chromosome. These calculations either pursue flat mode or vertical mode for accomplishing compression. But whichever method the existing genetic compression algorithms use, they are all based on searching similarities within the chromosome i.e. they exploit only inter chromosomal similarities. The current studies focus will show that compression ratio achieved by analyzing individual chromosome is always less than the method in which we analyze and compress intra chromosomal similarities.RESULTS: This study shows that by simply using exactly matching repeats amongst all the chromosomes of the same genome, not only the compression ratio is improving but also a detailed study of all the similarities and differences between two genomes of the same species can be conducted.CONCLUSION: In this study, a new compression algorithm is being proposed for compressing DNA. Along with Inter chromosomal similarities, Intra chromosomal similarities are considered for this method. The results clearly shows that intra chromosomal matches are bigger and more than inter chromosomal matches which helps us to achieve better compression ratio.

With the resolution enhancement techniques such as OPC (Optical Proximity Correction) and SRAF (Sub-Resolution Assist Features), the size of layout data have grown significantly. It is quite common now to find layout files that are tens of GBs in size. Unlike GDSII which can store data hierarchically, mask data formats such as MEBES are essentially flat and more voluminous. Moreover, polygonal data present in layout data files is fractured, thereby increasing the data volume before getting stored in MEBES data format. This results in huge MEBES files. As per the ITRS roadmap of 2005, for a 45nm half-pitch node that is expected to be in use by 2010, the mask data volume for a single layer is expected to reach up to 825 GB. Storing and transferring such large mask data are issues for which the mask industry needs solutions. Historically, MEBES is the most prevalent EB format in the industry. Moreover, in many Mask Data Preparation (MDP) flows, the MEBES format is being used as the de-facto standard for specifying the fractured EB data even though the final target EB machine might be different. In this paper we present techniques for lossless reversible compression of MEBES data, i.e., when the compressed file is decompressed, the generated uncompressed file matches the original MEBES file bit- by-bit. By applying these compression techniques a compression ratio of 5X to 15X can be obtained. In practice, compressing MEBES files is usually a one-time task, but decompression of compressed files is expected to be done multiple times as every time a compressed MEBES file needs processing, it has to be decompressed. MEBES is essentially an efficient data format and the geometries are stored compactly. As a result the compression/decompression techniques described in this paper are quite computation intensive in order to achieve higher compression ratio. This in turn leads to higher CPU time for compression/decompression compared to generic compressors such as gzip. However, as the format-specific compressors produce higher compression ratios, the disk I/O time for compression and decompression is expected to be less compared to the generic compressors such as gzip and gunzip. In spite of this, the format-specific decompressor is usually 3-5X times slower than the generic decompressor such as gunzip. Since decompression is expected to be done more frequently as compared to compression, speeding up decompression is highly desirable. We present a compression technique for MEBES data which enables multiple process threads to decompress the compressed MEBES data. With the multi-core multiprocessor machines becoming quite inexpensive and common, the multi-threaded decompression is expected to perform close to the disk I/O time on such machines. The paper details out the techniques, experimental results in terms of comparative compression ratios, compression and decompression speed using single threads and multiple threads. The possibility of getting higher compression ratios as well as higher or comparable decompression speed makes it more practical to use format specific reversible compression schemes rather than using generic compressors. Even though, the paper focuses on compression and decompression of MEBES, it can be easily extended to the compression of GDSII [1].

This paper presents a new lossless data compression method which can be used for the next-generation image and video codec. As an alternative system for CABAC (context adaptive binary arithmetic coder), a new binary coding algorithm is introduced. However, this method can be applicable to any kind of symbols. Efficiency of the codec in terms of compression ratio and computational complexity is introduced. Basic idea behind the proposed method is hiding data reversibly with a location map.

The rapid growth of smart grid systems has resulted in an exponential increase in data volume. This has led to significant challenges for efficient storage and transmission, thereby necessitating the development of advanced data compression techniques. Traditional techniques often struggle to maintain a balance between compression efficiency and data integrity, particularly when dealing with diverse and large datasets. To address this issue, this paper presents an adaptive data compression algorithm based on discrete wavelet transform (DWT) and Horse Herd Optimization (HHO). The proposed technique significantly improves storage by dynamically optimizing key performance metrics, such as signal-to-noise ratio (SNR), mean square error (MSE), and compression ratio (CR). By employing a robust optimization algorithm, it effectively addresses the computational challenges of processing large-scale and real-time data. This adaptability ensures the algorithm is well-suited for dynamic smart grid environments, providing a scalable and reliable solution to modern data management demands. This technique uses HHO to find the optimal thresholding for smart grid data compression. In general, DWT-based data compression is carried out using a universal threshold for ignoring particular wavelet coefficients. But, the performance of data compression varies for different threshold values. Hence, selecting an optimal threshold is a challenging task for data compression. Therefore, to solve this issue, an effective optimization algorithm is needed. In this work, a Multi-Objective Horse Herd Optimization (MO-HHO) algorithm is proposed to find the optimal threshold. The suggested MO-HHO algorithm accurately determines the global optimum threshold. Hence, it maintains a good compromise between SNR, MSE and CR. The effectiveness of the proposed algorithm is examined using three different data sets. Various datasets from the IEEE Power quality wave data, Household Electric Power Consumption data and Real-time dataset from an experimental set-up were used to test the proposed algorithm. The outcome demonstrates that the suggested MO-HHO algorithm performs better than the other conventional methods.

This paper presents a novel systems approach to compressing sensor network data. Unlike previous data compression methods, the proposed lossless linear predictor-based sensor data compression method utilizes structural system information to minimize the signal correlation in sensor network data. In the proposed method, linear predictor is derived in a system identification framework in which auto-regressive (AR) model is used as its model structure and the instrumental variables (IV) method is used to calculate the predictor parameters. A parametric study was carried out to study the effects of changes in system property, number of sensors, and sensor noise level on the compression performance of the proposed method. Both numerical simulation and experimental results show that the proposed sensor data compression method has a better compression performance than conventional linear predictor-based data compression method for single sensor.

Summary form only given. Large aperture static imaging spectrometer (LASIS) is a new kind of interferometer spectrometer with the advantages of high throughput and large field of view. The LASIS data contains spatial information in principle component along spatial direction and spectral information in modulated component along Optical Path Difference (OPD) direction. LASIS data have clearly parallel displacement of spatial information among continuous frames. It is a unique characteristic for LASIS data compared with other kinds of interferometer spectrometer such as SMII data and GITFS data. Although dispersive spectrometer, such as AVIRIS data, also contains some spatial information, each frame in a group shows the same ground view with different wavelengths spectrum without any motion displacement among inter frames. Based on these characteristics, we propose a lossless data compression method named Dual-direction Prediction Vector Quantization (DPVQ). With a dual-direction prediction on both spatial and spectral direction, redundancy in LASIS data is largely removed by minimizing the prediction residue in DPVQ. Then a fast vector quantization (VQ) avoiding codebook splitting process is applied after prediction. Considering time efficiency, the dual-direction prediction and VQ in DPVQ are optimized to reduce the calculations, so that optimized prediction saves 60% running time and fast VQ saves about 25% running time with a similar quantization quality compared with classical generalized Lloyd algorithm (GLA). Experimental results, shown in Fig. 2, indicate that DPVQ can achieve a maximal Compression Ratio (CR) at about 3.4, which outperforms many existing lossless compression algorithms.

Data compression techniques in IoT-enabled wireless body sensor networks: A systematic literature review and research trends for QoS improvement

Telemetric information is great in size, requiring extra room and transmission time. There is a significant obstruction of storing or sending telemetric information. Lossless data compression (LDC) algorithms have evolved to process telemetric data effectively and efficiently with a high compression ratio and a short processing time. Telemetric information can be packed to control the extra room and association data transmission. In spite of the fact that different examinations on the pressure of telemetric information have been conducted, the idea of telemetric information makes pressure incredibly troublesome. The purpose of this study is to offer a subsampled and balanced recurrent neural lossless data compression (SB-RNLDC) approach for increasing the compression rate while decreasing the compression time. This is accomplished through the development of two models: one for subsampled averaged telemetry data preprocessing and another for BRN-LDC. Subsampling and averaging are conducted at the preprocessing stage using an adjustable sampling factor. A balanced compression interval (BCI) is used to encode the data depending on the probability measurement during the LDC stage. The aim of this research work is to compare differential compression techniques directly. The final output demonstrates that the balancing-based LDC can reduce compression time and finally improve dependability. The final experimental results show that the model proposed can enhance the computing capabilities in data compression compared to the existing methodologies.

Suffix tree is a widely studied data structure with a range of applications. Although originally developed for string matching algorithms, it has proved to be useful for implementation of various lossless data compression methods. This interesting fact inspired us to explore the possibilities of creating new data compression methods, based entirely on properties of suffix trees. Resulting from a thorough study of existing suffix tree construction algorithms and their modifications, we propose and evaluate several variants of suffix tree based compression.KeywordsData CompressionActive PointSuffix TreeArithmetic CodeString Match AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

One of the goals of telemedicine is to enable remote visualization and browsing of medical volumes. There is a need to employ scalable compression schemes and efficient client-server models to obtain interactivity and an enhanced viewing experience. First, we present a scheme that uses JPEG2000 and JPIP (JPEG2000 Interactive Protocol) to transmit data in a multi-resolution and progressive fashion. The server exploits the spatial locality offered by the wavelet transform and packet indexing information to transmit, in so far as possible, compressed volume data relevant to the clients query. Once the client identifies its volume of interest (VOI), the volume is refined progressively within the VOI from an initial lossy to a final lossless representation. Contextual background information can also be made available having quality fading away from the VOI. Second, we present a prioritization that enables the client to progressively visualize scene content from a compressed file. In our specific example, the client is able to make requests to progressively receive data corresponding to any tissue type. The server is now capable of reordering the same compressed data file on the fly to serve data packets prioritized as per the client's request. Lastly, we describe the effect of compression parameters on compression ratio, decoding times and interactivity. We also present suggestions for optimizing JPEG2000 for remote volume visualization and volume browsing applications. The resulting system is ideally suited for client-server applications with the server maintaining the compressed volume data, to be browsed by a client with a low bandwidth constraint.

SummaryThe ability to design smarter, more predictive healthcare solutions for use in the community (at work and at home) and in healthcare facilities has been greatly enhanced by recent developments in the Internet of Health Things (IoHT) and cyber physical systems (CPS). These data collected by such medical sensors will be sent in a large quantity to the fog gateway of IoHT networks. These data should be forwarded to far cloud for further analysis and processing. Therefore, sending all of these data over the IoHT network to the data center will impose a significant burden on the IoHT network. This paper proposed a new lossless electroencephalogram (EEG) compression technique (NeLECoT) for fog computing‐based IoHT networks. It encodes the data of the patient at the fog gateway prior to sending it to the data center, thereby reducing the data's size. In the fog node, the NeLECoT combines three efficient techniques: clustering based on density‐based spatial clustering of applications with noise (DBSCAN), RLE (run length encoding), and Huffman encoding (HE). The clustering based on DBSCAN separates a massive volume of captured data into small groups of closely related (or similar) data. RLE encodes clustered EEG data, and the resulting file is encoded with HE. The fog gateway then transmits the encoded file to the cloud. Numerous simulation experiments were carried out, and the findings demonstrated that the suggested NeLECoT achieved better results than the competing techniques in terms of transmitted data size, compression ratio, compression power, compression time, decompression time, and average compression power.

The Internet of Things (IoT) end devices have major limitations related to hardware and energy autonomy. Generally, the highest energy consumption is related to communication, which accounts for up to 60% of consumption depending on the application. Among the strategies to optimize the energy consumed by communication, data compression methods are one of the most promising. However, most data compression algorithms are designed for personal computers and need to be adapted to the IoT context. This study aims to adapt classical algorithms, such as LZ77, LZ78, LZW, Huffman, and Arithmetic coding, and to analyse their performance and energy metrics in IoT end devices. The evaluation is performed in a device with an ESP32 processor and LoRa modulation. The study makes use of real datasets derived from two IoT applications. The results show compression rates close to 70%, a three-fold increase in the number of messages sent, and a reduction in energy consumption of 22%. An analytical model was also developed to estimate the gain in the battery life of the device using the adapted algorithms.

We use both lossy and lossless compression methods to reduce large amounts of digital data, making it easier to store and manage. Lossy and lossless compression is frequently used in astronomy, to condense the large amount of data collected by instruments namely, the James Webb Space Telescope and the Hubble Space Telescope. Similarly to diffraction images, astronomical images have large areas of empty space since stellar objects are very bright, yet show very small in images, which after a while, takes up a needless amount of limited digital storage space. Lossy compression achieves higher compression ratios but sacrifices some data quality in the process, while lossless compression preserves the data’s original form, but results in less storage savings. This work, conducted at Brookhaven National Laboratory’s National Synchrotron Light Source-II at the AMX and FMX macromolecular beam lines, aimed to evaluate the effectiveness of the lossy compression method known as Wavelet Compression. Wavelet compression is a method of data compression that uses wavelet transforms to reduce the size of data while preserving its essential features. Crystals are repetitive molecular assemblies that create diffraction patterns when exposed to X-rays, while detectors capture the amplitude of diffraction pattern spots (reflections). Similar to astronomical images, diffraction images contain large areas of empty space. This unnecessary data can quickly consume limited digital storage. By studying and optimizing compression methods, we not only improve data management but also advancements in various scientific disciplines.

An antidictionary is particularly useful for data compression, and on-line electrocardiogram (ECG) lossless compression algorithms using antidictionaries have been proposed. They work in real-time with constant memory and give better compression ratios than traditional lossless data compression algorithms, while they only deal with ECG data on a binary alphabet. This paper proposes on-line ECG lossless compression for a given data on a finite alphabet. The proposed algorithm gives not only better compression ratios than those algorithms but also uses less computational space than they do. Moreover, the proposed algorithm work in real-time. Its effectiveness is demonstrated by simulation results.