Multiple light field rendering

Light fields can be used to represent an object's appearance with a high degree of realism. However, unlike their geometric counterparts, these image-based representations lack user control for manipulating them. We present a system that allows a user to interactively manipulate, composite and render multiple light fields. LightShop is a modular system consisting of three parts: 1) a set of functions that allow a user to model a scene containing multiple light fields, 2) a ray-shading language that describes how an image should be constructed from a set of light fields, and 3) a real-time light field rendering system in OpenGL that can plug into existing 3D engines as a GLSL shader.We show applications in digital photography and we demonstrate how to integrate light fields into a modern space-flight game using LightShop.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Focal length inconsistency is a common problem in capturing multiple light fields (LFs), which reduces the accuracy in feature matching and LF registration, and limits the quality of LF stitching. In this paper, a novel method is proposed to stitch the LFs captured at different focal length accurately. First, refocus matching is proposed to find the best-matched-focused regions in the focal stack to improve the quality of feature matching. Then, depth distances and refocusing features are exploited to register central sub-aperture images (CSI) to handle focal length variation. Finally, CSI-based 4D warping is applied to preserve the LF angular consistency. Experimental results show that the proposed method outperforms the existing approaches obviously in terms of the stitched LFs and the coherence in epipolar plane image (EPI), especially for the LFs with the focal length inconsistency.

Light fields can provide transportive immersive experiences with a level of realism unsurpassed by any other imaging technology. Within a limited viewing volume, light fields accurately reproduce stereo parallax, motion parallax, reflections, refractions, and volumetric effects for real-world scenes. While light fields have been explored in computer graphics since the mid-90's [Gortier et al. 1996; Levoy and Hanrahan 1996], practical systems for recording, processing, and delivering high quality light field experiences have remained out of reach.In this talk, we describe the hardware and software developed to produce the Welcome to Light Fields virtual reality experience, the first immersive light field experience that can be downloaded and viewed on VR-ready computers with HTC Vive, Oculus Rift, and Windows Mixed Reality HMDs. This piece enables the user to step into a collection of panoramic light field still photographs and be guided through a number of light field environments, including the flight deck of Space Shuttle Discovery. Welcome to Light Fields is a free VR app. on the Steam Store that was launched in March 2018 [Google VR 2018].This piece is the culmination of a multi-year effort to develop consumer-ready light field technology. We built new camera rigs for light field capture and new algorithms for light field rendering. In this talk, we delve into the details from capture to rendering and the steps in between.We emphasize that our system generates stereoscopic views of a light field for use in current generation consumer VR HMDs. As future work, it could be of interest to display such datasets on near eye light field displays [Lanman and Luebke 2013] to render effects of visual accommodation.

We propose a load-balancing multi-LCD light field display technology. The multiple LCD panels operate as a spatial light modulator. Each light ray is the combination of pixels located in multiple LCD panels. The challenging problem is how to decompose the light field into limited layer images and display the light field compressively. Each pixel, as a controllable unit, is in spatial-multiplexing which means one pixel needs to be responsible to modulate multiple target light rays at the same time. We analyze the load imposed on each pixel by casting the light field decomposition as an over-determined equation problem. We found each pixel works in the state of overload and single pixel couldn’t give consideration to all target light rays. In order to reduce the load on pixels and improve display fidelity, we develop a multi-layer and multi-zone joint optimization strategy. The target light field is divided into multiple subzones and each subzone is displayed by multiple LCD panels combining with a dynamic directional backlight. By resolving the target light field, our display system further explores the multi-LCD’s capability of displaying light field and higher quality of light field display is achieved. We test our load-balancing decomposition algorithm based on different scene. The parallax, occlusion and blur of out-of-focus are restored successfully. And a three-layer prototype is constructed to demonstrate that correct light field is displayed in indoor lighting environment.

A light-field camera combines optics and computation to provide the ability to perform ranging. Many methods and algorithms, which are time-consuming and noise-sensitive, have been proposed for per-pixel depth estimation. We present a Fourier domain ranging method for light fields (LFs). Instead of estimating the depth for every pixel, we attempt to detect the depths of the different planes at which objects are located. The method is somewhat like using the energy in the focal stack, but it is carried out in a more efficient way. Our method has the advantages of speediness and robustness compared with traditional per-pixel depth estimation methods. In addition to the ranging algorithm, we also demonstrate the idea and application of a region adaptive denoising filter, in which the depth parameters are tuned by the proposed method. We include results for synthetic LFs, the Stanford LF archives, and LFs captured with a Lytro camera, exploiting the algorithm complexity, accuracy, depth resolution, and noise performance of our method. Our method uses less computation and is more robust than per-pixel depth estimation, making it appropriate for many applications, such as autorefocusing and autodenoising.

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Light field angular super-resolution based on intrinsic and geometric information

Light field rendering is an image-based rendering method that does not use 3D models but only images of the scene as input to render new views. Light field approximation, represented as a set of images, suffers from so-called refocusing artifacts due to different depth values of the pixels in the scene. Without information about depths in the scene, proper focusing of the light field scene is limited to a single focusing distance. The correct focusing method is addressed in this work and a real-time solution is proposed for focusing of light field scenes, based on statistical analysis of the pixel values contributing to the final image. Unlike existing techniques, this method does not need precomputed or acquired depth information. Memory requirements and streaming bandwidth are reduced and real-time rendering is possible even for high resolution light field data, yielding visually satisfactory results. Experimental evaluation of the proposed method, implemented on a GPU, is presented in this paper.

Sea ice pressure ridges have been recognized as important locations for both physical and biological processes. Thus, understanding the associated light‐field is crucial, but their complex structure and internal geometry render them hard to study by field methods. To calculate the in‐ and under‐ridge light field, we combined output from an ice mechanical model with a Monte‐Carlo ray tracing simulation. This results in realistic light fields showing that light levels within the ridge itself are significantly higher than under the surrounding level ice. Light guided through ridge cavities and scattering in between ridge blocks also results in a more isotropic ridge‐internal light field. While the true variability of light transmittance through a ridge can only be represented in ray tracing models, we show that simple parameterizations based on ice thickness and macro‐porosity allow accurate estimation of mean light levels available for photosynthesis underneath ridges in field studies and large‐scale models.

At the time of this paper, the advances in light field technology already offer 3D displays that immerse the users without the need for additional viewing devices. Despite the numerous advantages and attractive capabilities of such glasses-free 3D displays, their user interface methods are quite complicated and they are currently underwhelming when compared to conventional 2D displays, due to the fact that visual feedback can only be rendered sharply on the emission surface of light field displays. The sharp rendering of user interfaces is a necessity, as blur may hinder their fundamental functions. When it comes to 2D displays, many user interaction techniques and interfaces have been devised. Rendering a user interface on a 2D display could be done in various ways, such as rendering overlays on top of the rendered scene, or by using billboards. These are extensively used in modern video games. User interaction methods have proven their importance and added efficiency to virtual environments throughout the years. Due to their overall value and usefulness, interaction techniques develop immediately as new types of displays arise. With the recent advancements in visualization technologies, user interfaces have been redesigned for use in AR, VR and MR visualization. This includes on-screen augmentation, which enables interaction with visual content on the screen. Although light field displays contain immense potentials, only basic user interfaces have been devised thus far, including FOX (Focus Sliding surface), which grants users the option to scale and to rotate 3D objects. In this paper, we visualize the theater model on real light field displays and we test the different interactions by means of a monitor room. The theatre model is analogous to real-life theatres, where viewers may observe the theatrical presentation on the stage from various angles. The motivation to choose the theater model was the fact that light field visualization similarly allows multiple simultaneous viewers within its field of view, in which the content can be observed in an angle- dependent manner. Moreover, from the users’ perspective, the theater model is thus familiar and it provides high-quality visual feedback. Furthermore, theater stages encompass a lot of interactions, including rigging and flying systems, pulleys, rotating stages, lights, curtains etc. In order to test the different interaction methods on light field displays, a theater model depicting the virtual environment was implemented. Methods for rendering the monitor room and the results of the interactions are discussed in the paper, illustrated by images of the actual visualization on light field displays. It is shown that producing plausible results with no noticeable visual artifacts is challenging, yet possible. The scientific contributions of the paper also highlight the various novel user interfaces for future light field systems and services.

A method is proposed for depth extraction of low-texture region with a light field (LF) camera, which expands the application. Based on the analysis of LF data, it is proven that the depth information can be estimated from a single LF image. Furthermore, as the lenslet LF data can be decoded into a subimages array, and the relationship among subimages is proven to be affine transformation, we used the geometry relationship, which is represented by partition ratio of triangle grids area, to replace the unreliable gray value of low-texture region for stereo matching. In addition, to obtain accurate ratio values, preset points are projected to enrich the texture with a projector, which is convenient and reliable. The proposed method improves the accuracy of the depth extraction obviously at low-texture region compared with the traditional state-of-the-art LF method, and results are validated by experiments.

Tailored optical fields whose polarization and phase are coupled in a non-separable fashion usually referred to as vector beams, have become ubiquitous in a wide variety of research fields. These complex light fields can be generated through a combination of physical optical elements such as spiral phase plates, q-plates and quarter wave plates. However, such "hard-coded" optical devices present some restrictions such as their inability to generate multiple structured light fields simultaneously. This has stimulated a wide range of generation techniques based on computer controlled devices such as spatial light modulators (SLMs). SLMs provide on demand, fast and flexible holographic means to create multiple tailored optical fields simultaneously (multiplexing). Here we present a novel technique that utilizes the superposition principle, enabling digital generation of many vector beams on a single hologram. Unlike other techniques, this method is purely digital and allows the generation of a wide variety of vector beams by simply changing the encoded digital hologram. As a proof of concept we demonstrate simultaneous generation of vector beams with different polarization states on the High Order Poincaré Sphere (HOPS). This novel technique will be of impact in the context of quantum and classical communication, optical micro-manipulation and super resolution microscopy.

Light field sampling (LFS) theory can properly reduce minimum sampling rate while ensuring that novel views are not distorted for image-based rendering (IBR). The minimum sampling rate is determined by spectral support of light field. The spectral support of light field has studied the influence of the following factors: the minimum depth and the maximum depth, non-Lambertian reflections, whether the scene surfaces are flat, maximum frequency of painted signals. In this paper, we further perfect the light field spectrum analysis from the quantitative description of scene texture information based on the existing spectrum analysis theory. The quantification of texture information can be interactively refined via detected regional entropy. Thus, we can derive a spectral analytical function of light field with respect to texture information. The new function allows the spectral support of light field to be analyzed and estimated for different texture information associated with scene objects. In this way, we limit the spectral analysis problems of light field to those of a simpler signal. We show that this spectral analysis approach can be easily extended to arbitrary scene complexity levels, as we simplify the LFS of complex scenes to a plane. Additionally, the spectral support of light field broadens as the plane texture information becomes more complex. We present experimental results to demonstrate the performance of LFS with texture information, verify our theoretical analysis, and extend our conclusions on the optimal minimum sampling rate.

We propose a new method of rendering panoramic light field within a certain range, which provides an effective way for the acquisition of virtual reality data. Different from existing panoramic light field reconstruction algorithms, we propose the concept of the ray sphere to render a panoramic light field. Using the internal and external parameters of the cameras in the acquisition system, all the light field data are uniformly expressed in the ray sphere. The constructed ray sphere enables the rendering of a panoramic light field of any view point within a 3DOF+ space, which can not be achieved with correlation methods. In addition, we design and build an acquisition system to capture real scenes to verify the effectiveness of our method. Experimental results show that our method can get panoramic light field rendering of any view point on the horizontal plane within half the radius of the acquisition system, and can effectively process the light field video data.