Spatial pattern alterations from JPEG2000 lossy compression of remote sensing images: massive variogram analysis in high performance computing

<p>The projection into 2030 of the climate data volume increase brings an important challenge to the climate science community. This is particularly true for the CMIP7 that is projected to need about an Exabyte of storage capacity. Error-bounded lossy compression is explored as a potential solution to the above problem by different climate research teams. Several lossy compression schemes have been proposed leveraging different forms of decorrelation (transforms, prediction, HoSVD, DNN), quantization (linear, non-linear, vector), and encoding (dictionary-based, variable length, etc.) algorithms. Our experience with different applications shows that the compression methods often need to be customized and optimized to fit the specificities of the datasets to compress and the user requirements on the compression quality, ratio, and throughput. However, none of the existing lossy compression software for scientific data has been designed to be customizable. To address this issue, we developed SZ3, an innovative customizable, modular compression framework. SZ3 is a full C++ refactoring of SZ2 enabling the specialization, addition, or removal of each stage of the lossy compression pipeline to fit the specific characteristics of the datasets to compress and the use-case requirements. This extreme flexibility allows adapting SZ3 to many different use-cases, from ultra-high compression for visualization to ultra-high-speed compression between the CPU (or GPU) and the memory. Thanks to its unique set of features: customization, high compression ratio, high compression throughput, and excellent accuracy preservation, SZ3 won a 2021 R&D100 award. In this presentation, we present SZ3 and a new data prediction-based decorrelation method that significantly improves the compression ratios for climate datasets over the state-of-the-art lossy compressors, while preserving the same data accuracy. Experiments based on CESM datasets show that SZ3 can lead to up to 300% higher compression ratios than SZ2 with the same compression error bound and similar compression throughput.</p>

Image compression is the application of data compression techniques to two-dimensional digital images I(x, y) , to reduce the redundancy of the image data for storage or transmission in an efficient form. Image compression can be classified into two categories: lossless or lossy. Lossless compression, which achieves smaller compression ratios than lossy compression, mainly takes advantage of the image contents containing a non uniform probability distribution for a variable-length representation of the image pixels. Such images include technical drawings, icons or comics, and high-value contents such as medical imagery or image scans made for archival purposes. However, lossy compression methods, especially when they achieve a very high compression ratio, can introduce compression artifacts. Nevertheless, lossy compressions are especially suitable for natural images, such as photos, in applications where a minor (sometimes imperceptible) loss of fidelity is acceptable when it is desirable to achieve a substantial reduction in bitrate. Most of the stateof-the-art image compression standards use a combination of lossy and lossless algorithms to achieve the best performance. The Joint Photographic Experts Group (JPEG) [1], a discrete cosine transform(DCT)-based technique, is the most widely used standardized lossy image compression mechanism; it was designed for compressing either full-color or gray-scale images of natural, real-world, scenes. It works well for photographs, naturalistic artwork and similar material if the compression ratio is about 20 : 1, which is much better than the 4 : 1 compression ratio provided by a lossless compression method such as the Graphics Interchange Format (GIF) [2]. However, it does not work so well for lettering, simple cartoons, or line drawings.

To review the literature on lossy compression in dental radiography and to discuss the importance and suitability of the methodology used for evaluation of image compression. A search of Medline (from 1966 to October 2004) was undertaken with the search expression "(Radiography, dental) and compression". Inclusion criterion was that the reference should be evaluating the effect of lossy image compression on diagnostic accuracy. For all included studies, information in relation to mode of image acquisition, image content, image compression, image display, and method of image evaluation was extracted. 12 out of 32 papers were included in the review. The design of these 12 studies was found to vary considerably. Parameters used to express the degree of information loss (DIL) were either or both compression ratio (CR) and compression level (CL). The highest acceptable CR reported in the studies ranged from 3.6% to 15.4%. Furthermore, different CR values were proposed even for the same diagnostic task, for example, for caries diagnosis CR ranged from 6.2% to 11.1%. Lossy image compression can be used in clinical radiology if it does not conflict with national law. However, the acceptable DIL is difficult to express and standardize. CR is probably not suitable to express DIL, because it is image content dependent. CL is also probably not suitable to express DIL because of the lack of compression software standardization.

In this paper, we evaluate lossless and lossy compression techniques to compress quantitative phase images of red blood cells (RBCs) obtained by an off-axis digital holographic microscopy (DHM). The RBC phase images are numerically reconstructed from their digital holograms and are stored in 16-bit unsigned integer format. In the case of lossless compression, predictive coding of JPEG lossless (JPEG-LS), JPEG2000, and JP3D are evaluated, and compression ratio (CR) and complexity (compression time) are compared against each other. It turns out that JP2k can outperform other methods by having the best CR. In the lossy case, JP2k and JP3D with different CRs are examined. Because some data is lost in a lossy way, the degradation level is measured by comparing different morphological and biochemical parameters of RBC before and after compression. Morphological parameters are volume, surface area, RBC diameter, sphericity index, and the biochemical cell parameter is mean corpuscular hemoglobin (MCH). Experimental results show that JP2k outperforms JP3D not only in terms of mean square error (MSE) when CR increases, but also in compression time in the lossy compression way. In addition, our compression results with both algorithms demonstrate that with high CR values the three-dimensional profile of RBC can be preserved and morphological and biochemical parameters can still be within the range of reported values.

This paper deals with analysis of component-wise and three-dimensional (3D) lossy compression of hyperspectral data. The latter approach is known to be more efficient due to possibility to employ inter-band correlation of images in neighbor sub-bands. However, there are certain peculiarities for this approach which are subject of our study. Methods of lossy compression that employ discrete cosine transform (DCT) in blocks are studied. It is shown that 3D compression is able to provide considerable benefits in compression ratio and/or mean square error (MSE) of introduced distortions but only under certain conditions. Experiments have been done for real-life hyperspectral data with high input signal-to-noise ratio. Their main results are presented.

Alternative strategies used for wavelet-based lossy image compression can affect lesion detection differently at higher compression ratios. These effects were studied using three variants of a wavelet-based image compression algorithm: (1) unified quantization, (2) truncation of all coefficients at all subbands, and (3) truncation of coefficients subband by subband. The nonprewhitening- matched-filter-derived d<SUB>a</SUB>, a deductibility index, was used to quantify the changes in detection performance as a function of compression ratio for each strategy. Based on this approach, the optimal compression strategy was determined. Two classes of images were generated to simulate signal-present and signal-absent cases for a liver imaged by CT. For each strategy, the performance in discriminating between the signal-present class and signal-absent class was quantified by d<SUB>a</SUB> for varying compression ratios. Among the three strategies studied, truncation of all coefficients is the least desirable strategy for preserving small, low contrast signals; truncation of coefficients subband by subband yields the best result for subtle signals, but distorts high frequency edges between tissues; unified quantization is the best strategy if both low contrast objects and high frequency edges are to be preserved.

Vegetation indices (VIs) such as the Normalized Difference Vegetation Index (NDVI) are widely used for assessing vegetation cover and condition. One of the NDVI's significant disadvantages is its sensitivity to aerosols in the atmosphere, hence several atmospherically resistant VIs were formulated using the difference in the radiance between the blue and the red spectral bands. The state‐of‐the‐art atmospherically resistant VI, which is a standard Moderate Resolution Imaging Spectroradiometer (MODIS) product, together with the NDVI, is the Enhanced Vegetation Index (EVI). A different approach introduced the Aerosol‐free Vegetation Index (AFRI) that is based on the correlation between the shortwave infrared (SWIR) and the visible red bands. The AFRI main advantage is in penetrating an opaque atmosphere influenced by biomass burning smoke, without the need for explicit correction for the aerosol effect. The objective of this research was to compare the performance of these three VIs under smoke conditions. The AFRI was applied to the 2.1 µm SWIR channel of the MODIS sensor onboard the Earth Observing System (EOS) Terra and Aqua satellites in order to assess its functionality on these imaging platforms. The AFRI performance was compared with those of NDVI and EVI. All VIs were calculated on images with and without present smoke, using the surface‐reflectance MODIS product, for three case studies of fires in Arizona, California, and Zambia. The MODIS Fire Product was embedded on the images in order to identify the exact location of the active fires. Although good correlations were observed between all VIs in the absence of smoke (in the Arizona case R 2 = 0.86, 0.77, 0.88 for the NDVI–EVI, AFRI–EVI, and AFRI–NDVI, respectively) under smoke conditions a high correlation was maintained between the NDVI and the EVI, while low correlations were found for the AFRI–EVI and AFRI–NDVI (0.21 and 0.16, for the Arizona case, respectively). A time series of MODIS images recorded over Zambia during the summer of 2000 was tested and showed high NDVI fluctuations during the study period due to oscillations in aerosol optical thickness values despite application of aerosol corrections on the images. In contrast, the AFRI showed smoother variations and managed to better assess the vegetation condition. It is concluded that, beneath the biomass burning smoke, the AFRI is more effective than the EVI in observing the vegetation conditions.

The performance of diesel engine highly depends on atomization, vaporization and mixing of fuel with air. These factors are strongly influenced by various parameters e.g. injection pressure, injection timing, compression ratio, equivalence ratio, cylinder geometry, in cylinder air motion etc. In this study, a diesel engine has been investigated by employing a commercial CFD software (ANSYS Forte, version 18.1) especially developed for internal combustion engines (ICE) modeling; focusing primarily on the effects of equivalence ratio and compression ratio on combustion and emission characteristics. RNG k-ε model was employed as the turbulence model for analyzing the physical phenomena involved in the change of kinetic energy. In order to reduce the computational cost and time, a sector mesh of 45o angle with periodic boundary conditions applied at the periodic faces of the sector, is considered instead of using the whole engine geometry. Simulations are performed for a range of equivalence ratio varying from 0.6 to 1.2 and for three compression ratios namely, 15:1, 18:1 and 21:1. Results show that, improvement in combustion characteristics with higher compression ratio could be achieved for both lean and rich mixtures. Peak in-cylinder pressure and peak heat release nearer to TDC are achieved for compression ratio of 18:1 that could results in more engine torque. For compression ratio beyond 16:1, effects of fuel concentration on ignition delay is more pronounced. At lower compression ratio, in-cylinder temperature is not sufficiently high for atomization, vaporization, mixing of fuel with air, and preflame reactions to occur immediately after the fuel injection. NOx emission in diesel engine increases due to higher pressure and temperature inside the cylinder associated with relatively higher compression ratio. Rich mixture leads to more CO and unburnt hydrocarbon emission compared to lean mixture as result of incomplete combustion. Engine operation with too high compression ratio is detrimental as emission is a major concern.

A key milestone in NASA's Mission to Planet Earth project was achieved with the completion of Environmental Qualification testing of the Protoflight Model (PFM) Moderate Resolution Imaging Spectroradiometer (MODIS) instrument. Completing this task paved the way for MODIS to be integrated onto the EOS AM spacecraft which is to be launched as the keystone of the EOS system. Qualification of the PFM MODIS instrument required conducting an extensive test program in four different test facilities. Accomplishing environmental qualification testing, while meeting the stringent contamination and operational requirements for the MODIS instrument, required us to address a variety of issues and tasks. The main tasks included: developing special ground support test equipment, developing special tenting and handling equipment to protect the instrument from being contaminated during off-site environmental vibration an electromagnetic compatibility testing, designing and developing a state-of-the-art thermal-vacuum test chamber, and defining detailed test operations to fully characterize the instrument's electrical, optical and mechanical performance before, during and after each environmental test sequence. Selected penalty test were streamlined for characterizing the instrument whenever design changes or improved test techniques were incorporated to ensure all requirements had been met while maintaining a fully qualified instrument.

Integer-coefficient Discrete Wavelet Transformation (DWT) filters widely used in the literature are implemented and investigated as spectral decorrelator for on-board lossless hyperspectral image compression. As the performance of spectral decorrelation step has direct impact on the compression ratio (CR), it is important to employ the most convenient spectral decorrelator in terms of low computational complexity and high CR. Extensive tests using AVIRIS image data set are carried out and CRs corresponding to various subband decomposition levels are presented within a lossless hyperspectral compression framework. Results suggest that Cohen-Daubechies-Feauveau (CDF) 9/7 integer-coefficient wavelet transform with five levels of spectral subband decomposition would be an efficient spectral decorrelator for on-board lossless hyperspectral image compression.

This study proposes a unified LZ and hybrid coding (ULHC) method for compound image and video compression of visually lossless quality and high compression ratio. The method is macroblock‐based for ultra‐low coding latency and compatibility with the conventional and most popular hybrid video coding standards such as H.264 and MPEG‐2. First, each macroblock is coded by two tools: (i) gzip, a popular lossless LZ coding tool, modified to be macroblock‐oriented and to be seamlessly unifiable with a lossy hybrid coding tool; (ii) H.264, an advanced lossy hybrid coding tool. Then rate‐distortion optimisation is used to select either the modified gzip or H.264 as the final coding. To seamlessly unify these two coding tools for maximum quality and high compression ratio in ULHC, the modified gzip uses the most recent reconstructed and specially serialised macroblock data as the dictionary. Experimental results show that for images and videos composed of natural or synthesised picture, text and graphics, the proposed method provides higher peak signal‐to‐noise ratio and better subjective quality than H.264 at the same bitrate, and also achieves much higher compression ratio than gzip without any visual quality loss. In fact, ULHC can achieve partial‐lossless and partial‐near‐lossless coding with a high compression ratio.

In this paper, a progressive DCT-based coder (PDCTC), which implements a lossy image compression algorithm based on discrete cosine transform (DCT) and provides progressive data reconstruction, is introduced. It is compared to JPEG as well as to discrete atomic compression (DAC) that is an algorithm based on discrete atomic transform (DAT). It is shown that the algorithm proposed provides higher compression ratio (CR) than JPEG with the same quality loss measured by PSNR-metric. Also, it is proved that PDCTC and DAC have almost the same efficiency in terms of rate/distortion curve (dependence of PSNR and compression ratio). Besides, the complexity of these algorithms is analyzed.

3D point cloud presentation has been widely used in computer vision, automatic driving, augmented reality, smart cities and virtual reality. 3D point cloud compression method with higher compression ratio and tiny loss is the key to improve data transportation efficiency. In this paper, we propose a new 3D point cloud geometry compression method based on deep learning, also an auto-encoder performing better than other networks in detail reconstruction. It can reach much higher compression ratio than the state-of-art while keeping tolerable loss. It also supports parallel compressing multiple models by GPU, which can improve processing efficiency greatly. The compression process is composed of two parts. Firstly, Raw data is compressed into codeword by extracting feature of raw model with encoder. Then, the codeword is further compressed with sparse coding. Decompression process is implemented in reverse order. Codeword is recovered and fed into decoder to reconstruct point cloud. Detail reconstruction ability is improved by a hierarchical structure in our decoder. Latter outputs are grown from former fuzzier outputs. In this way, details are added to former output by latter layers step by step to make a more precise prediction. We compare our method with PCL compression and Draco compression on ShapeNet40 part dataset. Our method may be the first deep learning-based point cloud compression algorithm. The experiments demonstrate it is superior to former common compression algorithms with large compression ratio, which can also reserve original shapes with tiny loss.

In order to solve the problem of image blur identification caused by high compression ratio in unmanned aerial vehicle (UAV) airborne video images, a singular value decomposition (SVD) image compression method based on the region of interest (ROI) is proposed. Using regional compression method, the experiment firstly contour edge detection was carried out for images collected by UAV to help find out the key information part,and then divided the region of interest ROI on the high-frequency part, segmented it and compressed it to a small extent. Other areas were compressed according to the corresponding compression ratio according to the trans-mission bandwidth requirements, and finally the target area and background area were synthesized to obtain clear image of target with high compression ratio. A conclusion is obtained that the UAV can obtain a large peak signal to noise ratio (PSNR) with this method on the premise of guaranteeing the transmission of important information.

The problem of increasing the efficiency of coding of halftone images in the space of bit planes of differences in pixel values obtained using differential coding (DPCM – Differential pulse-code modulation) is considered. For a compact representation of DPCM pixel values, it is proposed to use a combined compression encoder that implements arithmetic coding and run-length coding. An arithmetic encoder provides high compression ratios, but has high computational complexity and significant encoding overhead. This makes it effective primarily for compressing the mean-value bit-planes of DPCM pixel values. Run-length coding is extremely simple and outperforms arithmetic coding in compressing long sequences of repetitive symbols that often occur in the upper bit planes of DPCM pixel values. For DPCM bit planes of pixel values of any image, a combination of simple run length coders and complex arithmetic coders can be selected that provides the maximum compression ratio for each bit plane and all planes in general with the least computational complexity. As a result, each image has its own effective combined encoder structure, which depends on the distribution of bits in the bit planes of the DPCM pixel values. To adapt the structure of the combined encoder to the distribution of bits in the bit planes of DPCM pixel values, the article proposes to use prediction of the volume of arithmetic code based on entropy and comparison of the obtained predicted value with the volume of run length code. The entropy is calculated based on the values of the number of repetitions of ones and zero symbols, which are obtained as intermediate results of the run length encoding. This does not require additional computational costs. It was found that in comparison with the adaptation of the combined encoder structure using direct determination of the arithmetic code volume of each bit plane of DPCM pixel values, the proposed encoder structure provides a significant reduction in computational complexity while maintaining high image compression ratios.