SADN: Learned Light Field Image Compression with Spatial-Angular Decorrelation

No-reference light field image quality assessment based on depth, structural and angular information

In this paper, we propose a new Light Field Image (LFI) compression scheme based on Multiview Video plus Depth (MVD) coding architecture. Through LF function analysis, we preliminarily estimate depth map according to the concept of Epipolar Plane Image (EPI). Such a rough estimation causes some error pixels within initial depth map, so we design a weighting mean filter to smooth the inaccurate region. The final estimated depth maps can be encoded by MVD coding standard jointly with a small number of viewpoint images in LFI, so as to improve compression efficiency of LFI. Ultimately, massive experiments are conducted on 4 LFIs to verify the effectiveness of the proposed compression scheme. The simulated results demonstrate that our LFI compression scheme can achieve a high LFI compression performance and outperform the-state-of-art coding solution.

The light field image, also known as the plenoptic image, contains the information about not only the intensity of light in a scene but also the direction of the light rays in space. Since the light field image contains very rich photometric and geometric information, it will have very widespread application in the future. For example, immersive content capture for virtual and mixed reality presentation or depth from light field for auto driving applications. To be more specific, a light field image can be enhanced with physical models for an autonomous decision making process, which is also an important task of the Dynamic Data Driven Applications Systems (DDDAS) [11]. Besides, the rich geometry and photometric information contained in the light field can be updated with real-time measurements, which is a focus of DDDAS such as smart city related image and video processing tasks. However, to make the light field images easier to be utilized, one of the most important tasks is to compress the light field images efficiently so they can be easily distributed over the current communication infrastructure.

Recent advances in light field technologies is fostering the research and development of novel imaging applications. Such applications perform processing of the light field information to create new visual effects such as, for example, refocusing, perspective change, colour adjustment. Light field imaging is very data intensive compared with usual digital photographic imaging, and novel compression algorithms are needed for addressing the problem of light field storage and transmission. Raw digital light field images exhibit low spatial correlation if compared to regular images and hence the performance in terms of rate-distortion of current state-of-the-art image encoders can be superseded by devising novel image compression architectures. In this paper, an architecture for lossy compression of unfocused light field images is proposed. Raw light fields are preprocessed by demosaicing, devignetting and slicing of the raw lenslet array image. The slices are then compressed with the JPEG 2000 image coding standard. The performance of the proposed method is compared against direct application of JPEG 2000 compression on the 4D light field. The experimental analysis has been conducted under a set of different compression ratios and the obtained results show that the proposed method outperforms direct application of the reference architecture.

View synthesis-based light field image compression using a generative adversarial network

Light field cameras are considered to have many potential applications since angular and spatial information is captured simultaneously. However, the limited spatial resolution has brought lots of difficulties in developing related applications and becomes the main bottleneck of light field cameras. In this paper, a learning-based method using residual convolutional networks is proposed to reconstruct light fields with higher spatial resolution. The view images in one light field are first grouped into different image stacks with consistent sub-pixel offsets and fed into different network branches to implicitly learn inherent corresponding relations. The residual information in different spatial directions is then calculated from each branch and further integrated to supplement high-frequency details for the view image. Finally, a flexible solution is proposed to super-resolve entire light field images with various angular resolutions. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods by a large margin in both visual and numerical evaluations. Furthermore, the proposed method shows good performances in preserving the inherent epipolar property in light field images.

Light field is the most prospective technology for interactive and immersive visual applications. and light field image is an intermediate data format that demands a large amount of storage space and higher transmission bandwidth. Therefore, compression of light field images is highly desired for further applications. In this paper, we propose a co-saliency guided bit allocation scheme with constraints of consistency among sub-aperture images. Firstly, saliency is jointly detected on color and depth images of sub-aperture by improving our previous model. The obtained pixel-wise co-saliency map is converted into block-wise via K-means clustering. In this way, the saliency weight of each coding tree unit (CTU) can be calculated. Then, target bits of each CTU are initially determined by the weight of each block. The allocation is adjusted dynamically under the guidance of co-saliency map and the image texture complexity. The experimental results show that BD-PSNR of 0.384 dB can be achieved for the salient region at the cost of less than 0.107 dB decrease for the whole image compared to HTM anchor. Moreover, subjective quality of proposed scheme outperforms the anchor for the salient region, and there is no noticeable distortion for non-salient region.

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

With the availability of commercial Light Field (LF) cameras, LF imaging has emerged as an up-and-coming technology in computational photography. However, the spatial resolution is significantly constrained in commercial micro-lens-based LF cameras because of the inherent multiplexing of spatial and angular information. Therefore, it becomes the main bottleneck for other applications of light field cameras. This paper proposes an adaptation module in a pre-trained Single Image Super-Resolution (SISR) network to leverage the powerful SISR model instead of using highly engineered light field imaging domain-specific Super Resolution models. The adaption module consists of a Sub-aperture Shift block and a fusion block. It is an adaptation in the SISR network to further exploit the spatial and angular information in LF images to improve the super-resolution performance. Experimental validation shows that the proposed method outperforms existing light field super-resolution algorithms. It also achieves PSNR gains of more than 1 dB across all the datasets as compared to the same pre-trained SISR models for scale factor 2, and PSNR gains 0.6 − 1 dB for scale factor 4.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

As light field images continue to increase in use and application, it becomes necessary to adapt existing image processing methods to this unique form of photography. In this paper we explore methods for applying neural style transfer to light field images. Feed-forward style transfer networks provide fast, high-quality results for monocular images, but no such networks exist for full light field images. Because of the size of these images, current light field data sets are small and are insufficient for training purely feed-forward style-transfer networks from scratch. Thus, it is necessary to adapt existing monocular style transfer networks in a way that allows for the stylization of each view of the light field while maintaining visual consistencies between views. To do this, we first generate disparity maps for each view given a single depth image for the light field. Then in a fashion similar to neural stylization of stereo images, we use disparity maps to enforce a consistency loss between views and to warp feature maps during the feed forward stylization. Unlike previous work, however, light fields have too many views to train a purely feed-forward network that can stylize the entire light field with angular consistency. Instead, the proposed method uses an iterative optimization for each view of a single light field image that backpropagates the consistency loss through the network. Thus, the network architecture allows for the incorporation of pre-trained fast monocular stylization network while avoiding the need for a large light field training set.

Currently, a raw light field image can be captured by a light field camera and decomposed into multi-views with different perceptual quality. A novel light field compression approach for a light field image is proposed based on this observation, where the raw light field image is decomposed into multi-views as pseudo-sequencing frames and reordered according to their perceptual quality. The resulting pseudo-temporal sequence is then compressed by high-efficiency video coding (HEVC) encoder. Experimental results show that the proposed method can achieve a better coding performance (up to 28% Bjontegaard delta bit rate reduction (BDBR) savings) compared with state-of-the-art reordering techniques using HEVC.

Recently, learning-based light field (LF) image compression methods have achieved impressive progress, while end-to-end spatially scalable LF image compression (SS-LFIC) has not been explored. To tackle this problem, this paper proposes an end-to-end spatially scalable LF compression network (SSLFC-Net). In the SSLFC-Net, a spatial-angular domain-specific enhancement layer coding strategy is designed to boost the coding performance of the enhancement layers (ELs). Specifically, by referencing domain-specific features, the ELs compress spatial features by predictive coding in the spatial domain to effectively remove inter-layer spatial redundancy, and reconstruct angular features by decoder-side generative method in the angular domain to strategically avoid angular compression. Particularly, to produce accurate spatial predictions and reconstruct high-quality LF images, an inter-layer spatial prediction module and a spatial-angular context-aware reconstruction module are presented to collaboratively promote EL compression. Experiments show that the proposed SSLFC-Net effectively supports spatial scalability and achieves state-of-the-art rate-distortion performance.

Diffraction-Assisted Light Field Microscopy for Microtomography and Digital Volume Correlation with Improved Spatial Resolution

Angular spectroscopy of light scattering can be used for quantitative analysis of cellular and subcellular properties, and thus promises a noninvasive methodology for in vivo assessment cellular integrity to complement in vitro histological examination. Spatial information is essential for accurate identification of localized abnormalities. However, conventional angular spectroscopy systems only provide single-channel measurement, which suffers from poor spatial resolution or requires time-consuming scanning over extended area. The purpose of this study was to develop a multi-channel angular spectroscopy for light field imaging in biological tissues. A microlens array (MLA) (8×8) based light field imager for 64-channel angular spectroscopy was developed. A pair of crossed polarizers was employed for polarization-sensitive recording to enable quantitative measurement at high signal specificity and sensitivity. The polarization-sensitive light field imager enables rapid measurement of multiple sampling volumes simultaneously at 18 μm spatial-resolution and 3° angular-resolution. Comparative light field imaging and electrophysiological examination of freshly isolated and physiologically deteriorated lobster leg nerves have been conducted. Two-dimensional (2D) polarization-sensitive scattering patterns of the fresh nerves were highly elliptical, while they gradually lost the ellipticity and became rotationally symmetric (i.e., circular) as the nerves physiologically deteriorated due to repeated electrical stimulations. Characterized parameters, i.e., the ellipticity and the scattering intensity, rendered spatially various characteristics such as different values and deteriorating rates. The polarization-sensitive light field imager is able to provide multi-channel angular spectroscopy of light scattering with both spatial and angular resolutions. The light scattering properties of nerves are highly dependent on the orientation of nerves and their physiological status. Further development of polarization-sensitive multi-channel angular spectroscopy may promise a methodology for rapid and reliable identification of localized abnormalities in biological tissues.