Real-time per-pixel focusing method for light field rendering

Light fields can provide transportive immersive experiences with a level of realism unsurpassed by any other imaging technology. Within a limited viewing volume, light fields accurately reproduce stereo parallax, motion parallax, reflections, refractions, and volumetric effects for real-world scenes. While light fields have been explored in computer graphics since the mid-90's [Gortier et al. 1996; Levoy and Hanrahan 1996], practical systems for recording, processing, and delivering high quality light field experiences have remained out of reach.In this talk, we describe the hardware and software developed to produce the Welcome to Light Fields virtual reality experience, the first immersive light field experience that can be downloaded and viewed on VR-ready computers with HTC Vive, Oculus Rift, and Windows Mixed Reality HMDs. This piece enables the user to step into a collection of panoramic light field still photographs and be guided through a number of light field environments, including the flight deck of Space Shuttle Discovery. Welcome to Light Fields is a free VR app. on the Steam Store that was launched in March 2018 [Google VR 2018].This piece is the culmination of a multi-year effort to develop consumer-ready light field technology. We built new camera rigs for light field capture and new algorithms for light field rendering. In this talk, we delve into the details from capture to rendering and the steps in between.We emphasize that our system generates stereoscopic views of a light field for use in current generation consumer VR HMDs. As future work, it could be of interest to display such datasets on near eye light field displays [Lanman and Luebke 2013] to render effects of visual accommodation.

Recently, light field cameras have drawn much attraction for their innovative performance in photographic and scientific applications. However, narrow baselines and constrained spatial resolution of current light field cameras impose restrictions on their usability. Therefore, we design a hybrid imaging system containing a light field camera and a high-resolution digital single lens reflex camera, and these two kinds of cameras share the same optical path with a beam splitter so as to achieve the reconstruction of high-resolution light fields. The high-resolution 4D light fields are reconstructed with a phase-based perspective variation strategy. First, we apply complex steerable pyramid decomposition on the high-resolution image from the digital single lens reflex camera. Then, we perform phase-based perspective-shift processing with the disparity value, which is extracted from the upsampled light field depth map, to create high-resolution synthetic light field images. High-resolution digital refocused images and high-resolution depth maps can be generated in this way. Furthermore, controlling the magnitude of the perspective shift enables us to change the depth of field rendering in the digital refocused images. We show several experimental results to demonstrate the effectiveness of our approach.

Light field angular super-resolution based on intrinsic and geometric information

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47×-7.28× speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

With the development of consumer light field cameras, the light field imaging has become an extensively used method for capturing the three-dimensional appearance of a scene. The depth estimation often requires a dense sampled light field in the angular domain or a high resolution in the spatial domain. However, there is an inherent trade-off between the angular and spatial resolutions of the light field. Recently, some studies for super-resolving the trade-off light field have been introduced. Rather than the conventional approaches that optimize the depth maps, these approaches focus on maximizing the quality of the super-resolved light field. In this article, we investigate how the depth estimation can benefit from these super-resolution methods. Specifically, we compare the qualities of the estimated depth using (a) the original sparse sampled light fields and the reconstructed dense sampled light fields, and (b) the original low-resolution light fields and the high-resolution light fields. Experiment results evaluate the enhanced depth maps using different super-resolution approaches.

A light field is a 4D function describing the radiance across a boundary between the volume containing a scene, and the disjoint volume in which the eyepoint may be placed. Light field rendering is the process of rendering novel views of a scene captured by the light field function. It is a purely image-based rendering technique which uses no geometric knowledge of the scene. Although the lack of needed geometric information make light fields an attractive way of capturing real-world scenes, it has made it difficult to effectively use light fields in combination with other rendering techniques, or even rendering multiple light fields in the same scene.We present a rendering method for light fields which allows multiple instances of a light field to be rendered in combination with a polygon rendered scene. Of particular note is our ability to render multiple light fields which appear to intersect in space consistently when viewed from any direction. The method is demonstrated by creating a forest consisting of thousands of instances of a tree light field, rendered on a polygon terrain.

Synthesizing a densely sampled light field from a single image is highly beneficial for many applications. Moreover, jointly solving both angular and spatial super-resolution problem also introduces new possibilities in light field imaging. The conventional method relies on physical-based rendering and a secondary network to solve the angular super-resolution problem. In addition, pixel-based loss limits the network capability to infer scene geometry globally. In this paper, we show that both super-resolution problems can be solved jointly from a single image by proposing a single end-to-end deep neural network that does not require a physical-based approach. Two novel loss functions based on known light field domain knowledge are proposed to enable the network to preserve the spatio-angular consistency between sub-aperture images. Experimental results show that the proposed model successfully synthesizes dense high resolution light field and it outperforms the state-of-the-art method in both quantitative and qualitative criteria. The method can be generalized to arbitrary scenes, rather than focusing on a particular subject. The synthesized light field can be used for various applications, such as depth estimation and refocusing.

Light fields are an image-based representation that represents objects using sets of digital images. Light fields are usually comprised of 4D radiance data that can be used for 3D rendering or 4D display on autostereoscopic and certain types of volumetric displays. We present a modeling and rendering system for spherical isotropic light fields. These are made of images captured with cameras placed on a sphere's surface looking inwards. The system implements a multiresolution representation for both the spatial and the directional domains of the light field. This representation supports continuous levels of detail and adaptive frame rate control.

We propose a hybrid computational near-eye light field display based on micro-lens arrays (MLAs) and structure parameters prioritization method (SPPM) that can render correct accommodation and expand the depth of field (DOF). The proposed display consists of two MLAs and two micro-displays. Two sets of depth-sliced images are superimposed to reconstruct the object's light field. The system's ray resolution and the spectra difference between the proposed light field and the conventional one are discussed. The DOF of the proposed system is extended without the requiring of high refresh rate display. Furthermore, SPPM is proposed to extend the spatial frequency of the light field. A prototype is demonstrated and the experimental result shows that the depth field of the reconstructed images is significantly increased compared with conventional near-eye light field display based on single MLA.

A light-field camera combines optics and computation to provide the ability to perform ranging. Many methods and algorithms, which are time-consuming and noise-sensitive, have been proposed for per-pixel depth estimation. We present a Fourier domain ranging method for light fields (LFs). Instead of estimating the depth for every pixel, we attempt to detect the depths of the different planes at which objects are located. The method is somewhat like using the energy in the focal stack, but it is carried out in a more efficient way. Our method has the advantages of speediness and robustness compared with traditional per-pixel depth estimation methods. In addition to the ranging algorithm, we also demonstrate the idea and application of a region adaptive denoising filter, in which the depth parameters are tuned by the proposed method. We include results for synthetic LFs, the Stanford LF archives, and LFs captured with a Lytro camera, exploiting the algorithm complexity, accuracy, depth resolution, and noise performance of our method. Our method uses less computation and is more robust than per-pixel depth estimation, making it appropriate for many applications, such as autorefocusing and autodenoising.

A method is presented for isolating moving distractors from a static background in imagery captured by a hovering or slowly moving Autonomous Underwater Vehicle (AUV). By reparameterizing a set of monocular images into a light field structure, it becomes possible to apply a linear fan filter and its inverse to extract the background and distractors, respectively. Results are shown for a hovering AUV imaging a region with non-trivial 3D structure and containing moving elements. The output is a distractor-free 3D light field model of the sea floor and a set of images of isolated distractors. We show that the technique is insensitive to parallax in the background elements, outperforming pixel differencing techniques.

Although face recognition is a widely accepted form of biometrics due to its ease of capture, it still suffers from out-of-focus imaging due to the limited depth-of-field found in conventional 2D imaging systems. In order to overcome such a challenge, light field cameras have been successfully employed in biometrics. The number of depth images obtained from a light field camera can be optimally used by fusing a number of these images or selecting the best focus image from the set of images. In this work, we employ a method to pre-select the images when more than two images corresponding to different focus are present. Further, we evaluate various fusion techniques against the all − in − focus image obtained from light field imaging for the verification performance in face recognition systems. The evaluation of various schemes is performed on one of the largest light field face databases consisting of 80 subjects. The best Equal Error Rate (EER) of 4.14% is observed with the combination of employed pre-selection method and Laplacian Pyramid based fusion approach using both sparse representation and multi-scale transform.

Light field rendering (LFR) is an image-based method for generating novel views from a set of camera images. When sample of camera array is sparse, conventional light field rendering would cause aliasing artifacts. We present a straightforward layer-based method that significantly removes the aliasing artifacts with no prior knowledge of the scene geometry. First, a sparse disparity map is generated to detect the depth layers of the scene. Then, for synthesized each pixel, using the color consistency constraint to detect the optimum focal plane from the layer depths, and rendering such pixel with selected focal plane by LFR method. Finally, a trick method is used to reduce flickering artifacts caused by the color consistency constraint. Experimental results show the effectiveness of our approach.

We present a novel Geometry-aware Neural Interpolation (Geo-NI) framework for light field rendering. Previous learning-based approaches either perform direct interpolation via neural networks, which we dubbed Neural Interpolation (NI), or explore scene geometry for novel view synthesis, also known as Depth Image-Based Rendering (DIBR). Both kinds of approaches have their own strengths and weaknesses in addressing non-Lambert effect and large disparity problems. In this paper, we incorporate the ideas behind these two kinds of approaches by launching the NI within a specific DIBR pipeline. Specifically, a DIBR network in the proposed Geo-NI serves to construct a novel reconstruction cost volume for neural interpolated light fields sheared by different depth hypotheses. The reconstruction cost can be interpreted as an indicator reflecting the reconstruction quality under a certain depth hypothesis, and is further applied to guide the rendering of the final high angular resolution light field. To implement the Geo-NI framework more practically, we further propose an efficient modeling strategy to encode high-dimensional cost volumes using a lower-dimension network. By combining the superiorities of NI and DIBR, the proposed Geo-NI is able to render views with large disparities with the help of scene geometry while also reconstructing the non-Lambertian effect when depth is prone to be ambiguous. Extensive experiments on various datasets demonstrate the superior performance of the proposed geometry-aware light field rendering framework.

Sea ice pressure ridges have been recognized as important locations for both physical and biological processes. Thus, understanding the associated light‐field is crucial, but their complex structure and internal geometry render them hard to study by field methods. To calculate the in‐ and under‐ridge light field, we combined output from an ice mechanical model with a Monte‐Carlo ray tracing simulation. This results in realistic light fields showing that light levels within the ridge itself are significantly higher than under the surrounding level ice. Light guided through ridge cavities and scattering in between ridge blocks also results in a more isotropic ridge‐internal light field. While the true variability of light transmittance through a ridge can only be represented in ray tracing models, we show that simple parameterizations based on ice thickness and macro‐porosity allow accurate estimation of mean light levels available for photosynthesis underneath ridges in field studies and large‐scale models.