Efficient Depth Estimation for Light Field Images

No-reference light field image quality assessment based on depth, structural and angular information

Depth estimation and camera calibration of a focused plenoptic camera for visual odometry

View synthesis-based light field image compression using a generative adversarial network

Robust depth estimation on real-world light field images using Gaussian belief propagation

Methods based on light field information have shown promising results in depth estimation and underwater image restoration. However, improvements are still needed in terms of depth estimation accuracy and image restoration quality. Previous work on underwater image restoration employed an image formation model (IFM) that overlooked the effects of light attenuation and scattering coefficients in underwater environments, leading to unavoidable color deviation and distortion in the restored images. Additionally, the high blurriness and associated distortions in underwater images make depth information extraction and estimation very challenging. In this paper, we refine the light propagation model and propose a method to estimate the attenuation and backscattering coefficients of the underwater IFM. We simplify these coefficients into distance-related functions and design a relationship between distance and the darkest channel to estimate the water coefficients, effectively suppressing color deviation and distortion in the restoration results. Furthermore, to increase the accuracy of depth estimation, we propose using blur cues to construct a cost for refocusing in the depth direction, reducing the impact of high signal-to-noise ratio environments on depth information extraction, and effectively enhancing the accuracy and robustness of depth estimation. Finally, experimental comparisons show that our method achieves more accurate depth estimation and image restoration closer to real scenes compared to state-of-the-art methods.

In this paper, we propose a new Light Field Image (LFI) compression scheme based on Multiview Video plus Depth (MVD) coding architecture. Through LF function analysis, we preliminarily estimate depth map according to the concept of Epipolar Plane Image (EPI). Such a rough estimation causes some error pixels within initial depth map, so we design a weighting mean filter to smooth the inaccurate region. The final estimated depth maps can be encoded by MVD coding standard jointly with a small number of viewpoint images in LFI, so as to improve compression efficiency of LFI. Ultimately, massive experiments are conducted on 4 LFIs to verify the effectiveness of the proposed compression scheme. The simulated results demonstrate that our LFI compression scheme can achieve a high LFI compression performance and outperform the-state-of-art coding solution.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Perceptual encryption has received widespread attention as a technology for protecting multimedia visual information. In multimedia applications, perceptual encryption only protects a portion of the content without previewing all the information. At present, perceptual encryption is mostly oriented to conventional plain images while few methods aim at visual security measures for perceptually encrypted images. Existing solutions usually adopt well-known quality assessment metrics to measure the visual quality of encrypted images. However, they often exhibit undesired behavior on perceptually encrypted images with low quality. As a typical representation of three-dimensional scenes, light field images record the intensity and direction of light during propagation which is distinct from 2-D images. In this paper, we construct a perceptually encrypted light field image database (PE-LFID) for quality assessment based on 14 reference light field plaintext images. For each scene, we employ four encryption methods, each of which has six levels. Additionally, a novel visual security evaluation method based on PE-LFID is proposed by taking into account the local and global features of the light field images. First, we use a multi-threshold edge detection method to obtain the edge similarity in the spatial domain of the light field image. Afterwards, the epipolar plane image (EPI) generated from the angular domain of the light field image is used to calculate the gradient magnitude similarity, which is expressed as a global feature. Furthermore, the final quality prediction score is calculated by adaptively weighting between local and global features. We conduct extensive experiments on the proposed PE-LFID to assess the performance of classical and state-of-the-art IQA models. The experimental results demonstrate the effectiveness of the proposed method for visual security evaluation of perceptually encrypted light field images, as well as the scalability of PE-LFID.

The Light field (LF) imaging technology can obtain the spatial information and angular information of light simultaneously. Since, the sensor captures multiple images from different angular directions, the spatial resolution of each view image is limited. LF image super-resolution (SR) methods based on deep-learning have been proposed to solve this problem. However, correlation of LF image at two angular dimensions have not been fully exploited in algorithms that based on epipolar plane images (EPIs). In this paper, we propose a joint LF image SR network by combining a multiple magnification single image SR (SISR) network and an inpainting network for EPI solids. The EPI solids, which contain information of both angular dimensions, are extracted from the output of SISR network and used as input data of the inpainting network. The inpainting network uses channel attention mechanism in feature extraction module to extracting line and gradient feature contained by EPI solids and correlations between EPI slices of EPI sold for restoring the geometric continuity of different views. Experimental results show that the proposed multiple magnification spatial LF image SR network which is trained by dataset with mixed magnifications has better performance than other approaches that are specially trained at each magnification.

In this paper, an aberration correction method for an extended target is proposed to solve the problem of the lenslet-based plenoptic camera not imaging clearly under the influence of aberrations. We propose a light field manipulation method to improve performance of the light field imaging system. The principle of this method is that the sub-aperture images extracted from the raw light field image are offset when the light field imaging system is affected by aberrations, and the symmetrical arrangement of the sub-aperture image array is destroyed. By repairing the symmetrical arrangement of the sub-aperture image array, the influence of phase aberrations on the imaging system can be eliminated, and the resolution of the plenoptic camera can be improved. We use an image correlation algorithm to process the sub-aperture images of the plenoptic camera, calculate and compensate each sub-aperture image's displacement caused by aberrations, and restore the symmetrical arrangement of the sub-aperture image array; then, a corrected high-resolution refocused image can be generated. In particular, this method uses only the raw light field information obtained by the plenoptic camera in a single exposure, without adding other hardware devices. Furthermore, it takes the extended target itself as the reference image, so the ideal position need not be calibrated in advance. Also, the parallax information of the sub-aperture images is retained, and the method is simple and easy to use. Numerical simulation and experimental results show that the technology proposed in this paper can work well for high-resolution imaging of a plenoptic camera with phase aberrations. This method can be potentially applied to analyze lens aberration, media-induced image distortion such as water turbulence in underwater imaging, and atmospheric turbulence in remote imaging. It may have important application prospects in the fields of astronomical object detection, remote sensing, etc.

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

Learning an epipolar shift compensation for light field image super-resolution

Synthesizing a densely sampled light field from a single image is highly beneficial for many applications. Moreover, jointly solving both angular and spatial super-resolution problem also introduces new possibilities in light field imaging. The conventional method relies on physical-based rendering and a secondary network to solve the angular super-resolution problem. In addition, pixel-based loss limits the network capability to infer scene geometry globally. In this paper, we show that both super-resolution problems can be solved jointly from a single image by proposing a single end-to-end deep neural network that does not require a physical-based approach. Two novel loss functions based on known light field domain knowledge are proposed to enable the network to preserve the spatio-angular consistency between sub-aperture images. Experimental results show that the proposed model successfully synthesizes dense high resolution light field and it outperforms the state-of-the-art method in both quantitative and qualitative criteria. The method can be generalized to arbitrary scenes, rather than focusing on a particular subject. The synthesized light field can be used for various applications, such as depth estimation and refocusing.

Acquisition of densely-sampled light fields (LFs) is challenging. In this paper, we develop a coarse-to-fine light field angular superresolution that reconstructs densely-sampled LFs from sparsely-sampled ones. Unlike most of other methods, which are limited by the regularity of sampling patterns, our method can flexibly deal with different scale factors with one model. Specifically, a coarse restoration on epipolar plane images (EPIs) with arbitrary angular resolution is performed and then a refinement with 3D convolutional neural networks (CNNs) on stacked EPIs. The subaperture images in LFs are synthesized first horizontally, then vertically, forming a pseudo 4DCNN. In addition, our method can handle large baseline light field without using geometry information, which means it is not constrained by Lambertian assumption. Experimental results over various light field datasets including large baseline LFs demonstrate the significant superiority of our method when compared with state-of-the-art ones.

With the development of consumer light field cameras, the light field imaging has become an extensively used method for capturing the three-dimensional appearance of a scene. The depth estimation often requires a dense sampled light field in the angular domain or a high resolution in the spatial domain. However, there is an inherent trade-off between the angular and spatial resolutions of the light field. Recently, some studies for super-resolving the trade-off light field have been introduced. Rather than the conventional approaches that optimize the depth maps, these approaches focus on maximizing the quality of the super-resolved light field. In this article, we investigate how the depth estimation can benefit from these super-resolution methods. Specifically, we compare the qualities of the estimated depth using (a) the original sparse sampled light fields and the reconstructed dense sampled light fields, and (b) the original low-resolution light fields and the high-resolution light fields. Experiment results evaluate the enhanced depth maps using different super-resolution approaches.