High-spatial-resolution light field 3D perception based on unidirectional object motion

Light Field, a computational photography technique has various exciting features like refocusing and novel view synthesis. Previous works in light field photography have focused their work on creating light field images having a limited field of view. This limitation of light field images motivates the research for a acquisition system which can capture light field images with larger field of view. Recent works have been done to acquire light fields using a cylindrical arrangement of cameras which resolve the conventional complexity issue of acquisition of light field and subsequently merging them by essentially combining the capture and merging processes into a single exercise. We use this cylindrical system to generate a novel True Zoomed Light field panorama from the captured light field, which gives a wide 360? field of view similar to traditional panoramas along with the zoom applied to the whole panorama. The captured panorama can also be manipulated in various ways as is possible with other light field recordings. Also various zooming options which can be easily found in traditional camera, either tends to change the relative shape of the objects in image or introduces pix-elation effect. We use true zoom so as to improvise the zooming effect. We generate novel “True Zoomed Light Field Panorama” using our system. This type of panorama gives user a feeling of being closer to or farther from the whole environment at the same time.

With the availability of commercial Light Field (LF) cameras, LF imaging has emerged as an up-and-coming technology in computational photography. However, the spatial resolution is significantly constrained in commercial micro-lens-based LF cameras because of the inherent multiplexing of spatial and angular information. Therefore, it becomes the main bottleneck for other applications of light field cameras. This paper proposes an adaptation module in a pre-trained Single Image Super-Resolution (SISR) network to leverage the powerful SISR model instead of using highly engineered light field imaging domain-specific Super Resolution models. The adaption module consists of a Sub-aperture Shift block and a fusion block. It is an adaptation in the SISR network to further exploit the spatial and angular information in LF images to improve the super-resolution performance. Experimental validation shows that the proposed method outperforms existing light field super-resolution algorithms. It also achieves PSNR gains of more than 1 dB across all the datasets as compared to the same pre-trained SISR models for scale factor 2, and PSNR gains 0.6 − 1 dB for scale factor 4.

Ear recognition is an emerging research area in image-based biometrics. The commercial availability of lenslet light field cameras able to capture full spatio-angular information has brought momentum to biometric and forensic research exploiting this new type of imaging sensors. This study is the first to consider the usage of light field cameras for ear recognition, proposing both an appropriate content database and an ear recognition solution. The novel Lenslet Light Field Ear DataBase (LLFEDB) includes 536 light field images corresponding to four different poses, from 67 subjects, captured with a Lytro ILLUM lenslet light field camera. The LLFEDB includes critical cases such as ear images partly occluded by ear piercing, earing, hair and combination of occlusions. The novel light field-based ear recognition solution proposed exploits the richer spatio-angular information available in the light field images. A comparative performance evaluation study using the LLFEDB, and focusing on the most recent light field and non-light field based descriptors for ear recognition, shows a very promising performance of the proposed descriptor, outperforming all the assessed descriptors.

Light field cameras are considered to have many potential applications since angular and spatial information is captured simultaneously. However, the limited spatial resolution has brought lots of difficulties in developing related applications and becomes the main bottleneck of light field cameras. In this paper, a learning-based method using residual convolutional networks is proposed to reconstruct light fields with higher spatial resolution. The view images in one light field are first grouped into different image stacks with consistent sub-pixel offsets and fed into different network branches to implicitly learn inherent corresponding relations. The residual information in different spatial directions is then calculated from each branch and further integrated to supplement high-frequency details for the view image. Finally, a flexible solution is proposed to super-resolve entire light field images with various angular resolutions. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods by a large margin in both visual and numerical evaluations. Furthermore, the proposed method shows good performances in preserving the inherent epipolar property in light field images.

In this paper, an aberration correction method for an extended target is proposed to solve the problem of the lenslet-based plenoptic camera not imaging clearly under the influence of aberrations. We propose a light field manipulation method to improve performance of the light field imaging system. The principle of this method is that the sub-aperture images extracted from the raw light field image are offset when the light field imaging system is affected by aberrations, and the symmetrical arrangement of the sub-aperture image array is destroyed. By repairing the symmetrical arrangement of the sub-aperture image array, the influence of phase aberrations on the imaging system can be eliminated, and the resolution of the plenoptic camera can be improved. We use an image correlation algorithm to process the sub-aperture images of the plenoptic camera, calculate and compensate each sub-aperture image's displacement caused by aberrations, and restore the symmetrical arrangement of the sub-aperture image array; then, a corrected high-resolution refocused image can be generated. In particular, this method uses only the raw light field information obtained by the plenoptic camera in a single exposure, without adding other hardware devices. Furthermore, it takes the extended target itself as the reference image, so the ideal position need not be calibrated in advance. Also, the parallax information of the sub-aperture images is retained, and the method is simple and easy to use. Numerical simulation and experimental results show that the technology proposed in this paper can work well for high-resolution imaging of a plenoptic camera with phase aberrations. This method can be potentially applied to analyze lens aberration, media-induced image distortion such as water turbulence in underwater imaging, and atmospheric turbulence in remote imaging. It may have important application prospects in the fields of astronomical object detection, remote sensing, etc.

Light field imaging is an emerging technology in computational photography areas. Based on innovative designs of the imaging model and the optical path, light field cameras not only record the spatial intensity of threedimensional (3D) objects, but also capture the angular information of the physical world, which provides new ways to address various problems in computer vision, such as 3D reconstruction, saliency detection, and object recognition. In this paper, three key aspects of light field cameras, i.e., model, calibration, and reconstruction, are reviewed extensively. Furthermore, light field based applications on informatics, physics, medicine, and biology are exhibited. Finally, open issues in light field imaging and long-term application prospects in other natural sciences are discussed.

Specular reflection removal is indispensable to many computer vision tasks. However, most existing methods fail or degrade in complex real scenarios for their individual drawbacks. Benefiting from the light field imaging technology, this paper proposes a novel and accurate approach to remove specularity and improve image quality. We first capture images with specularity by the light field camera (Lytro ILLUM). After accurately estimating the image depth, a simple and concise threshold strategy is adopted to cluster the specular pixels into “unsaturated” and “saturated” category. Finally, a color variance analysis of multiple views and a local color refinement are individually conducted on the two categories to recover diffuse color information. Experimental evaluation by comparison with existed methods based on our light field dataset together with Stanford light field archive verifies the effectiveness of our proposed algorithm.

Currently, a raw light field image can be captured by a light field camera and decomposed into multi-views with different perceptual quality. A novel light field compression approach for a light field image is proposed based on this observation, where the raw light field image is decomposed into multi-views as pseudo-sequencing frames and reordered according to their perceptual quality. The resulting pseudo-temporal sequence is then compressed by high-efficiency video coding (HEVC) encoder. Experimental results show that the proposed method can achieve a better coding performance (up to 28% Bjontegaard delta bit rate reduction (BDBR) savings) compared with state-of-the-art reordering techniques using HEVC.

As light field images continue to increase in use and application, it becomes necessary to adapt existing image processing methods to this unique form of photography. In this paper we explore methods for applying neural style transfer to light field images. Feed-forward style transfer networks provide fast, high-quality results for monocular images, but no such networks exist for full light field images. Because of the size of these images, current light field data sets are small and are insufficient for training purely feed-forward style-transfer networks from scratch. Thus, it is necessary to adapt existing monocular style transfer networks in a way that allows for the stylization of each view of the light field while maintaining visual consistencies between views. To do this, we first generate disparity maps for each view given a single depth image for the light field. Then in a fashion similar to neural stylization of stereo images, we use disparity maps to enforce a consistency loss between views and to warp feature maps during the feed forward stylization. Unlike previous work, however, light fields have too many views to train a purely feed-forward network that can stylize the entire light field with angular consistency. Instead, the proposed method uses an iterative optimization for each view of a single light field image that backpropagates the consistency loss through the network. Thus, the network architecture allows for the incorporation of pre-trained fast monocular stylization network while avoiding the need for a large light field training set.

Recently, light field cameras have drawn much attraction for their innovative performance in photographic and scientific applications. However, narrow baselines and constrained spatial resolution of current light field cameras impose restrictions on their usability. Therefore, we design a hybrid imaging system containing a light field camera and a high-resolution digital single lens reflex camera, and these two kinds of cameras share the same optical path with a beam splitter so as to achieve the reconstruction of high-resolution light fields. The high-resolution 4D light fields are reconstructed with a phase-based perspective variation strategy. First, we apply complex steerable pyramid decomposition on the high-resolution image from the digital single lens reflex camera. Then, we perform phase-based perspective-shift processing with the disparity value, which is extracted from the upsampled light field depth map, to create high-resolution synthetic light field images. High-resolution digital refocused images and high-resolution depth maps can be generated in this way. Furthermore, controlling the magnitude of the perspective shift enables us to change the depth of field rendering in the digital refocused images. We show several experimental results to demonstrate the effectiveness of our approach.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Augmented reality(AR) technology is becoming the study focus, and the AR effect of the light field imaging makes the research of light field camera attractive. The array structure was adopted in most light field information acquisition system(LFIAS) since emergence of light field camera, lens array(MLA) and pinhole array(MPA) system mainly included. It is reviewed in this paper the structure of the LFIAS that the Light field camera commonly used in recent years. LFIAS has been analyzed based on the theory of geometrical optics. Meanwhile, this paper presents a novel LFIAS, plane grating system, we call it micro aperture array(MAA. And the LFIAS are analyzed based on the knowledge of information optics; This paper proves that there is a little difference in the multiple image produced by the plane grating system. And the plane grating system can collect and record the amplitude and phase information of the field light.

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

<p>Light field photography offers a new approach to digitally captured images. These commercially available cameras are able to capture the 4D light field in a single image. This allows for a variety of image processing capabilities that traditional cameras do not offer. For example, the image can be digitally refocused after it is captured and its depth can be estimated. In terms of application, these capabilities could be beneficial on airborne platforms. However, a limitation of currently available light field cameras is that they are not fully functional at medium or long ranges. If these cameras were to capture light fields at longer ranges, they would have a practical application when mounted on low-flying aircrafts. This dissertation takes current light field photography techniques and modifies them so they work better to capture medium-range images. The majority of cameras that capture the 4D light field use a microlens array to modulate the incoming light before it hits the image sensor. Previous work using printed modulation masks garnered the same effect obtained by microlens arrays. This dissertation details the development of a modulation mask that has medium-range applications. A new way of extracting the 4D light field from raw images that uses a digital Fourier transform is presented. This method works for images captured with microlens arrays and printed mask cameras. Two prototype cameras were built and tested to demonstrate some of these concepts. The concepts demonstrated by these cameras could be used in the future designs of light field cameras.</p>

Light field images taken by plenoptic cameras often have a tradeoff between spatial and angular resolutions. In this paper, we propose a novel spatial super-resolution approach for light field images by jointly exploiting internal and external similarities. The internal similarity refers to the correlations across the angular dimensions of the 4D light field itself, while the external similarity refers to the cross-scale correlations learned from an external light field dataset. Specifically, we advance the classic projection-based method that exploits the internal similarity by introducing the intensity consistency checking criterion and a back-projection refinement, while the external correlation is learned by a CNN-based method which aggregates all warped high-resolution sub-aperture images upsampled from the low-resolution input using a single image super-resolution method. By analyzing the error distributions of the above two methods and investigating the upperbound of combining them, we find that the internal and external similarities are complementary to each other. Accordingly, we further propose a pixel-wise adaptive fusion network to take advantage of both their merits by learning a weighting matrix. Experimental results on both synthetic and real-world light field datasets validate the superior performance of the proposed approach over the state-of-the-arts.