Visual Quality Assessment for Perceptually Encrypted Light Field Images

No-reference light field image quality assessment based on depth, structural and angular information

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Blind light field image quality assessment by analyzing angular-spatial characteristics

Light field images can record multiple information about the light rays in a scene and provide multiple views from a single image, offering a new data source for 3D reconstruction. However, ensuring the quality of light field images themselves is challenging, and distorted image inputs may lead to poor reconstruction results. Accurate light field image quality assessment can pre-judge the quality of light field images used as input for 3D reconstruction, providing a reference for the reconstruction results before the reconstruction work, significantly improving the efficiency of 3D reconstruction based on light field images. In this paper, we propose an Adaptive Vision Transformer-based light-field image-quality assessment model (AViT-LFIQA). The model adopts a multi-view sub-aperture image sequence input method, greatly reducing the number of input images while retaining as much information as possible from the original light field image, alleviating the training pressure on the neural network. Furthermore, we design an adaptive learnable attention layer based on ViT, which addresses the lack of inductive bias in ViT by using adaptive diagonal masking and a learnable temperature coefficient strategy, making the model more suitable for training on small datasets of light field images. Experimental results demonstrate that the proposed model is effective for various types of distortions and shows superior performance in light-field image-quality assessment.

In this paper, we propose a new Light Field Image (LFI) compression scheme based on Multiview Video plus Depth (MVD) coding architecture. Through LF function analysis, we preliminarily estimate depth map according to the concept of Epipolar Plane Image (EPI). Such a rough estimation causes some error pixels within initial depth map, so we design a weighting mean filter to smooth the inaccurate region. The final estimated depth maps can be encoded by MVD coding standard jointly with a small number of viewpoint images in LFI, so as to improve compression efficiency of LFI. Ultimately, massive experiments are conducted on 4 LFIs to verify the effectiveness of the proposed compression scheme. The simulated results demonstrate that our LFI compression scheme can achieve a high LFI compression performance and outperform the-state-of-art coding solution.

The Light field (LF) imaging technology can obtain the spatial information and angular information of light simultaneously. Since, the sensor captures multiple images from different angular directions, the spatial resolution of each view image is limited. LF image super-resolution (SR) methods based on deep-learning have been proposed to solve this problem. However, correlation of LF image at two angular dimensions have not been fully exploited in algorithms that based on epipolar plane images (EPIs). In this paper, we propose a joint LF image SR network by combining a multiple magnification single image SR (SISR) network and an inpainting network for EPI solids. The EPI solids, which contain information of both angular dimensions, are extracted from the output of SISR network and used as input data of the inpainting network. The inpainting network uses channel attention mechanism in feature extraction module to extracting line and gradient feature contained by EPI solids and correlations between EPI slices of EPI sold for restoring the geometric continuity of different views. Experimental results show that the proposed multiple magnification spatial LF image SR network which is trained by dataset with mixed magnifications has better performance than other approaches that are specially trained at each magnification.

The light field image, also known as the plenoptic image, contains the information about not only the intensity of light in a scene but also the direction of the light rays in space. Since the light field image contains very rich photometric and geometric information, it will have very widespread application in the future. For example, immersive content capture for virtual and mixed reality presentation or depth from light field for auto driving applications. To be more specific, a light field image can be enhanced with physical models for an autonomous decision making process, which is also an important task of the Dynamic Data Driven Applications Systems (DDDAS) [11]. Besides, the rich geometry and photometric information contained in the light field can be updated with real-time measurements, which is a focus of DDDAS such as smart city related image and video processing tasks. However, to make the light field images easier to be utilized, one of the most important tasks is to compress the light field images efficiently so they can be easily distributed over the current communication infrastructure.

Light field images consist of a set of sub-aperture images with slight shifts in their positions. They can be used to obtain the depth map without camera calibration and image rectification. However, since the baseline between sub-aperture images is very narrow, it limits the depth estimation range and accuracy. In this paper, we propose a depth estimation method to solve such problems. After we analyze the sub-aperture images in the form of epipolar plane images (EPIs), we modify the edge confidence measure to test which part of EPIs is good for depth estimation. Then, we apply the modified structure tensor (MST) to compute depth and corresponding information. Finally, we reconstruct a depth map from light field images.

Splitting the backbone: A novel hierarchical method for assessing light field image quality

Light filed imaging is becoming popular for its diversity of post-processing and a wide range of applications. Various kinds of research about light field such as light field compression methods are coming out one after the other in recent years. For better evaluation of the quality of light field images and the performance of compression algorithm, the study on quality assessment of light field is in desperate need. In this paper, in order to establish a light field quality assessment database for the subsequent research, we propose a methodology of subjective evaluation for light field image and use a 2D objective evaluation method to verify the methodology. Results show that this methodology can be successfully used to assess the quality of light field content.

Learning an epipolar shift compensation for light field image super-resolution

Robust depth estimation on real-world light field images using Gaussian belief propagation

Angular spectroscopy of light scattering can be used for quantitative analysis of cellular and subcellular properties, and thus promises a noninvasive methodology for in vivo assessment cellular integrity to complement in vitro histological examination. Spatial information is essential for accurate identification of localized abnormalities. However, conventional angular spectroscopy systems only provide single-channel measurement, which suffers from poor spatial resolution or requires time-consuming scanning over extended area. The purpose of this study was to develop a multi-channel angular spectroscopy for light field imaging in biological tissues. A microlens array (MLA) (8×8) based light field imager for 64-channel angular spectroscopy was developed. A pair of crossed polarizers was employed for polarization-sensitive recording to enable quantitative measurement at high signal specificity and sensitivity. The polarization-sensitive light field imager enables rapid measurement of multiple sampling volumes simultaneously at 18 μm spatial-resolution and 3° angular-resolution. Comparative light field imaging and electrophysiological examination of freshly isolated and physiologically deteriorated lobster leg nerves have been conducted. Two-dimensional (2D) polarization-sensitive scattering patterns of the fresh nerves were highly elliptical, while they gradually lost the ellipticity and became rotationally symmetric (i.e., circular) as the nerves physiologically deteriorated due to repeated electrical stimulations. Characterized parameters, i.e., the ellipticity and the scattering intensity, rendered spatially various characteristics such as different values and deteriorating rates. The polarization-sensitive light field imager is able to provide multi-channel angular spectroscopy of light scattering with both spatial and angular resolutions. The light scattering properties of nerves are highly dependent on the orientation of nerves and their physiological status. Further development of polarization-sensitive multi-channel angular spectroscopy may promise a methodology for rapid and reliable identification of localized abnormalities in biological tissues.

How to improve the rendering efficiency and realism of light field images remains a major challenge. This paper proposes a rendering approach to efficiently generate photorealistic light field images. Firstly, the next-generation model optimization technique is used to transform the complex high-polygon model (HPM) into the low-polygon model (LPM). Then, the rendering speed is accelerated by using the LPM and geometric perspective coherence. Next, the normals of the HPM are baked onto the LPM using model baking techniques, and the high-precision light field images are rendered using LPM and the normals of HPM. This process can reduce 95% of the model data, thereby increasing the rendering speed by more than 10 times, while maintaining the rendering quality comparable to the HPM. Secondly, nearly half of the triangles are further decreased by using a two-step backface culling technique, thereby doubling the rendering speed again. The rendering speed is greatly improved using the GPU-CPU hybrid architecture. Thirdly, to enhance realism, anisotropic illumination effects are realized by using the perspective coherence of lighting, and vivid shadows are rendered utilizing an image-space shadow mapping technology. The results indicate that the proposed method can increase the rendering speed by more than 20 times and maintain the image's SSIMs exceeding 0.9 and PSNRs above 30 dB. Finally, a light field display system based on integral imaging was built, and high-resolution and attractive 3D images with a wide viewing angle were synthesized using the light field images and display system.