Compression of interferometric radio-astronomical data

Hyperspectral image compression has become increasingly important in data exploitation because of enormous data volumes and high redundancy provided by hundreds of contiguous spectral channels. Since a hyperspectral image can be viewed as a 3-dimensional (3D) image cube, many efforts have been devoted to extending 2D image compression techniques to perform 3D image compression on hyperspectral image cubes. Unfortunately, some major issues generally encountered in hyperspectral data exploitation at low or very low-bit rate compression, for example, subpixels and mixed pixels which do not occur in traditional pure pixel-based image compression are often overlooked in such a 2D-to-3D compression. Accordingly, a direct application of 2D-to-3D compression techniques to hyperspectral image cubes without taking precaution may result in significant loss of crucial spectral information provided by subtle substances such as small objects, anomalies during low bit-rate lossy compression. This paper takes a rather different view by investigating lossy hyperspectral compression from a perspective of exploring spectral information, referred to as exploitation-based lossy compression and further develops spectral/spatial hyperspectral image compression to effectively preserve crucial and vital spectral information of objects which are generally missed by commonly used mean-squared error (MSE) or signal-to-noise ratio (SNR)-based compression techniques when lossy compression is performed at low bit rates. In order to demonstrate advantages of the proposed spectral/spatial compression approach applications of subpixel target detection and mixed pixel analysis are used for experiments for performance evaluation.

Today’s scientific simulations generate exceptionally large volumes of data, challenging the capacities of available I/O bandwidth and storage space. This necessitates a substantial reduction in data volume, for which error-bounded lossy compression has emerged as a highly effective strategy. A crucial metric for assessing the efficacy of lossy compression is visualization. Despite extensive research on the impact of compression on visualization, there is a notable gap in the literature concerning the effects of compression on the visualization of Adaptive Mesh Refinement (AMR) data. AMR has proven to be a potent solution for addressing the rising computational intensity and the explosive growth in data volume that requires storage and transmission. However, the hierarchical and multi-resolution characteristics of AMR data introduce unique challenges to its visualization, and these challenges are further compounded when data compression comes into play. This article delves into the intricacies of how data compression influences and introduces novel challenges to the visualization of AMR data.

Modern Earth modeling systems often use high-resolution unstructured grids to discretize their horizontal domains. One of the major challenges in working with these high-resolution models is to efficiently transmit and store large volumes of model data for operational forecasts and for modeling research.A newly developed compression technique is presented that significantly reduces the size of datasets produced by high-resolution global models that are discretized on an icosahedral grid. The compression technique is based on the wavelet transform together with a grid rearrangement algorithm and precision-controlled quantization technology. The grid rearrangement algorithm converts an icosahedral grid to a set of 10 rhombus grids that retain the spatial correlation of model data so that a three-dimensional wavelet transform can be effectively applied. The precision-controlled quantization scheme guarantees specified precision of compressed datasets.The technique is applied to the output of a global weather prediction model, the Flow-Following, Finite-Volume Icosahedral Model (FIM) developed by NOAA’s Earth System Research Laboratory. Experiments show that model data at 30-km resolution can be compressed up to 50:1 without noticeable visual differences; at specified precision requirements, the proposed compression technique achieves better compression compared to a state-of-the-art compression format [Gridded Binary (GRIB) with JPEG 2000 packing option]. In addition, model forecasts initialized with original and compressed initial conditions are compared and assessed. The assessment indicates that it is promising to use the technique to compress model data for those applications demanding high fidelity of compressed datasets.

To review the literature on lossy compression in dental radiography and to discuss the importance and suitability of the methodology used for evaluation of image compression. A search of Medline (from 1966 to October 2004) was undertaken with the search expression "(Radiography, dental) and compression". Inclusion criterion was that the reference should be evaluating the effect of lossy image compression on diagnostic accuracy. For all included studies, information in relation to mode of image acquisition, image content, image compression, image display, and method of image evaluation was extracted. 12 out of 32 papers were included in the review. The design of these 12 studies was found to vary considerably. Parameters used to express the degree of information loss (DIL) were either or both compression ratio (CR) and compression level (CL). The highest acceptable CR reported in the studies ranged from 3.6% to 15.4%. Furthermore, different CR values were proposed even for the same diagnostic task, for example, for caries diagnosis CR ranged from 6.2% to 11.1%. Lossy image compression can be used in clinical radiology if it does not conflict with national law. However, the acceptable DIL is difficult to express and standardize. CR is probably not suitable to express DIL, because it is image content dependent. CL is also probably not suitable to express DIL because of the lack of compression software standardization.

The current investigation experimentally studied the effects of compression on the acoustic performance of porous fibrous material. Two inch and four inch thick samples of fiberglass and three varying densities of mineral wool were tested using two different impedance tube sizes at compression rates of 1, 1.3 and 2. The absorption coefficient was measured using Chung and Blaser’s method. The flow resistivity was measured using Tao et al.’s method. Overall, the 4” samples resulted in steadier results than the 2” samples. Compression generally led to a decrease in absorption coefficient and an increase in flow resistivity. These effects were most evident in the lower frequency range. Although there were some experimental errors in sample preparation, sample variation, compression technique, testing order and other initial errors, the current study demonstrated that the effects of compression on insulation should be not be overlooked.

Pruned-tree structured vectored quantization (PTSVQ) was applied to the lower five gray scale remapped bits of normal and fatty ultrasound liver images. The upper bits were compressed reversibly. This combination of techniques is termed PTSVQ with splitting. The effect of the compression on the difference in texture between normal and fatty liver images was studied at different compression rates and distortions. The changes in texture were measured by changes in the principal components of the covariance matrix of image vectors. The vectors were the same size as those used in the compression technique. There were clear differences in the components of normal and fatty liver images. These differences were largely removed by the PTSVQ with splitting technique even at average single pixel distortions several times smaller than the image noise. These results suggest that the effect of compression on second order statistics should be measured when evaluating algorithms in addition to the first order average distortion.

In the scientific domain, extremely large amounts of data are generated by large-scale high performance computing (HPC) simulations. Storing and sending such vast volumes of data poses serious scalability and performance issues, which can be considerably mitigated by data compression techniques which significantly reduced storage size and data movement burdens. Since scientific data are being shared by scientists more and more frequently, data security methods that ensure the confidentiality, integrity, and availability of data itself are becoming increasingly important. As such, combing compression and encryption is critical to storing large-scale datasets securely. In this work, we explore how to integrate data compression and cryptography techniques as efficiently as possible for big scientific datasets in the HPC field. We perform thorough experiments using different scientific datasets with the state-of-the-art error-bounded lossy compressor - SZ - on a real-world supercomputing environment. Experiments verify that performing encryption before lossy compression (a.k.a., encr-cmpr method) may invalidate the advantage of compression algorithms. By contrast, executing encryption after lossy compression (a.k.a., cmpr-encr method) keeps not only high compression ratios but high overall execution speed. Experiments also verify that the encryption overhead under the cmpr-encr method decreases with increasing compression ratios, which means very good scalability.

43 - Volume Visualization in Medicine

Chapter 1 - Introduction

The ability of technology to transmit multi-media is very dependent on compression techniques. In particular lossy compression has been used in image compression (jpeg) audio compression (mp3) and video compression (mpg) to allow the transmission of audio and video over broadband network connections. Recently the sense of touch or haptics is becoming more important with its addition in computer games or in cruder applications such as vibrations in a cell phone. As haptic technology improves the ability to transmit compressed force sensations becomes more critical. Most lossy audio and visual compression techniques rely on the lack of sensitivity in humans to pick up detailed information in certain scenarios. Similarly limitations in the sensitivity of human touch could be exploited to create haptic models with much less detail and thus requiring smaller bandwidth. The focus of this paper is on the force thresholds of the human haptic system that can be used in a psychophysically motivated lossy haptic (force) compression technique. Most of the research in this field has measured the just noticeable difference (JND) of the human haptic system with a human user in static interaction with a stationary rigid object. In this paper our focus involves cases where the human user or the object are in relative motion. An example of such an application would be the haptic rendering of the user's hand in contact with of a high-viscous material or interacting with a highly deformable object. Thus an approach is presented to measure the force threshold based on the velocity of the user's hand motion. Two experiments are conducted to detect the absolute force threshold (AFT) of the human haptic system using methodologies from the field of psychophysics. The AFTs are detected for three different ranges of velocity of the user's hand motion. This study implies that when a user's hand is in motion fewer haptic details are required to be stored calculated or transmitted. Finally the implications of this study on a more complete future study will be discussed.

State-of-the-art graphic processing units (GPUs) provide very high memory bandwidth, but the performance of many general-purpose GPU (GPGPU) workloads is still bounded by memory bandwidth. Although compression techniques have been adopted by commercial GPUs, they are only used for compressing texture and color data, not data for GPGPU workloads. Furthermore, the microarchitectural details of GPU compression are proprietary and its performance benefits have not been previously published. In this paper, we first investigate required microarchitectural changes to support lossless compression techniques for data transferred between the GPU and its off-chip memory to provide higher effective bandwidth. Second, by exploiting some characteristics of floating-point numbers in many GPGPU workloads, we propose to apply lossless compression to floating-point numbers after truncating their least-significant bits (i.e., lossy compression). This can reduce the bandwidth usage even further with very little impact on overall computational accuracy. Finally, we demonstrate that a GPU with our lossless and lossy compression techniques can improve the performance of memory-bound GPGPU workloads by 26% and 41% on average.

Hyperspectral data lossy compression has not yet achieved global acceptance in the remote sensing community, mainly because it is generally perceived that using compressed images may affect the results of posterior processing stages. This possible negative effect, however, has not been accurately characterized so far. In this letter, we quantify the impact of lossy compression on two standard approaches for hyperspectral data exploitation: spectral unmixing, and supervised classification using support vector machines. Our experimental assessment reveals that different stages of the linear spectral unmixing chain exhibit different sensitivities to lossy data compression. We have also observed that, for certain compression techniques, a higher compression ratio may lead to more accurate classification results. Even though these results may seem counterintuitive, this work explains these observations in light of the spatial regularization and/or whitening that most compression techniques perform and further provides recommendations on best practices when applying lossy compression prior to hyperspectral data classification and/or unmixing.

The cost of I/O is a significant challenge on current supercomputers, and the trend is likely to continue into the foreseeable future. This challenge is amplified in scientific visualization because of the requirement to consume large amounts of data before processing can begin. Lossy compression has become an important technique in reducing the cost of performing I/O. In this paper we consider the implications of using compressed data for visualization within a scientific workflow. We use visualization operations on simulation data that is reduced using three different state-of-the-art compression techniques. We study the storage efficiency and preservation of visualization features on the resulting compressed data, and draw comparisons between the three techniques used. Our contributions can help inform both scientists and researchers in the use and design of compression techniques for preservation of important visualization details.

A novel Adaptive Lossy Image Compression (ALIC) technique is proposed to achieve high compression ratio by reducing the number of source symbols through the application of an efficient technique. The proposed algorithm is based on processing the discrete cosine transform (DCT) of the image to extract the highest energy coefficients in addition to applying one of the novel quantization schemes proposed in the present work. This method is straightforward and simple. It does not need complicated calculation; therefore the hardware implementation is easy to attach. Experimental comparisons are carried out to compare the performance of the proposed technique with those of other standard techniques such as the JPEG. The experimental results show that the proposed compression technique achieves high compression ratio with higher peak signal to noise ratio than that of JPEG at low bit rate without the visual degradation that appears in case of JPEG.

The results of a detailed theoretical study on collinear guided wave to leaky wave acoustooptic (AO) interactions in proton-exchanged LiNbO(3) (PE:LiNbO(3)) planar waveguides are presented. The guided-to-leaky mode conversion for an input optical beam at the wavelength of 632.8 nm by the induced diffraction grating from a collinear surface acoustic wave (SAW) is analyzed using a generalized multimode formulation of the coupled mode theory. Mode conversion efficiency and AO bandwidth have been calculated as functions of acoustic frequency, interaction length, guiding layer thickness, and acoustic drive power density for three cuts of the LiNbO(3) substrate. High performance configurations that are desirable for application to demultiplexing and switching in optical communication systems are identified, and the corresponding channel capacity and frequency resolution are determined. For example, it was shown that the X-cut configuration features the highest mode conversion efficiency. However, a relatively small AO bandwidth is associated with this configuration. Both high mode conversion efficiency and large AO bandwidth can be accomplished at the guiding layer thickness of 1.0 microm. A TM(o)-->TE(nu) mode conversion efficiency as high as 42% together with an AO bandwidth of approximately 70 MHz can be achieved in the Z-cut waveguide at the guiding layer thickness of 1.0 microm, acoustic drive power density of 50 mW/mm, interaction length of 40 mm, and acoustic frequency of 460 MHz. The corresponding channel capacity and frequency resolution are 745 and 0.09 MHz, respectively. Measured mode conversion efficiencies as high as 90 and 78% obtained at the acoustic frequencies of 107 and 367 MHz using the X-cut substrate and the Y-propagation SAW have verified the theoretical prediction on the mode conversion efficiencies.