Real-Time Light Field Processing for Autonomous Robotics

In this paper, an aberration correction method for an extended target is proposed to solve the problem of the lenslet-based plenoptic camera not imaging clearly under the influence of aberrations. We propose a light field manipulation method to improve performance of the light field imaging system. The principle of this method is that the sub-aperture images extracted from the raw light field image are offset when the light field imaging system is affected by aberrations, and the symmetrical arrangement of the sub-aperture image array is destroyed. By repairing the symmetrical arrangement of the sub-aperture image array, the influence of phase aberrations on the imaging system can be eliminated, and the resolution of the plenoptic camera can be improved. We use an image correlation algorithm to process the sub-aperture images of the plenoptic camera, calculate and compensate each sub-aperture image's displacement caused by aberrations, and restore the symmetrical arrangement of the sub-aperture image array; then, a corrected high-resolution refocused image can be generated. In particular, this method uses only the raw light field information obtained by the plenoptic camera in a single exposure, without adding other hardware devices. Furthermore, it takes the extended target itself as the reference image, so the ideal position need not be calibrated in advance. Also, the parallax information of the sub-aperture images is retained, and the method is simple and easy to use. Numerical simulation and experimental results show that the technology proposed in this paper can work well for high-resolution imaging of a plenoptic camera with phase aberrations. This method can be potentially applied to analyze lens aberration, media-induced image distortion such as water turbulence in underwater imaging, and atmospheric turbulence in remote imaging. It may have important application prospects in the fields of astronomical object detection, remote sensing, etc.

Light field reconstruction from images captured by focal plane sweeping, such as light field moment imaging~(LFMI) and light field reconstruction with back projection~(LFBP), can achieve high transverse resolution comparable to the modern camera sensor. This is impossible for the conventional microlens-array-based light field imaging systems. However, the acquisition of a series of focal plane sweeping images along the optical axis is time consuming and requires fine alignment. Besides, different focal plane based light field reconstruction techniques require images with different characteristics. To solve these problems, we present an efficient approach for fast light field acquisition with precise focal plane sweeping capture by defocus modulation rather than mechanical movement. Because of the controllable point spread function, we can capture images for light field reconstruction with both LFMI and LFBP

The image quality might be degraded severely due to water scatterings, especially near-shore water scatterings. In addition, the degraded image reduces the accuracy of underwater 3D reconstruction. We propose an underwater imaging system based on light field camera array to obtain enhanced high-quality images and accurate three-dimensional reconstruction from harsh underwater environments. Both 2D-imaging and 3D-reconstruction using light-field imaging were studied by experiments. By refocusing and analyzing the light field images of the target through turbid water, we obtained accurate three-dimensional estimations in the underwater scene. These experiments confirm the reliability of our method, showing that the SSIM (structural similarity) value of the system in turbid water is 0.378, compared to 0.197, the SSIM value of conventional single-camera imaging in the experiments. Therefore, we verify that the light field imaging system is less affected by forward scattering of water and has a larger detective range than conventional single-camera imaging with any specific resolution.

Light field (LF) imaging has gained significant attention due to its recent success in microscopy, 3-dimensional (3D) displaying and rendering, augmented and virtual reality usage. Postprocessing of LF enables us to extract more information from a scene compared to traditional cameras. However, the use of LF is still a research novelty because of the current limitations in capturing high-resolution LF in all of its four dimensions. While researchers are actively improving methods of capturing high-resolution LF's, using simulation, it is possible to explore a high-quality captured LF's properties. The immediate concerns following the LF capture are its storage and processing time. A rich LF occupies a large chunk of memory ---order of multiple gigabytes per LF---. Also, most feature extraction techniques associated with LF postprocessing involve multi-dimensional integration that requires access to the whole LF and is usually time-consuming. Recent advancements in computer processing units made it possible to simulate realistic images using physical-based rendering software. In this work, at first, a transformation function is proposed for building a camera array (CA) to capture the same portion of LF from a scene that a standard plenoptic camera (SPC) can acquire. Using this transformation, LF simulation with similar properties as a plenoptic camera will become trivial in any rendering software. Artificial intelligence (AI) and machine learning (ML) algorithms ---when deployed on the new generation of GPUs--- are faster than ever. It is possible to generate and train large networks with millions of trainable parameters to learn very complex features. Here, residual convolutional neural network (RCNN) structures are employed to build complex networks for compression and feature extraction from an LF. By combining state-of-the-art image compression and RCNN, I have created a compression pipeline. The proposed pipeline's bit per pixel (bpp) ratio is 0.0047 on average. I show that with a 1% compression time cost and 18x speedup for decompression, our methods reconstructed LFs have better structural similarity index metric (SSIM) and comparable peak signal-to-noise ratio (PSNR) compared to the state-of-the-art video compression techniques used to compress LFs. In the end, using RCNN, I created a network called RefNet, for extracting a group of 16 refocused images from a raw LF. The training parameters of the 16 LFs are set to (\alpha=0.125, 0.250, 0.375, ..., 2.0) for training. I show that RefNet is 134x faster than the state-of-the-art refocusing technique. The RefNet is also superior in color prediction compared to the state-of-the-art

Light fields provide a natural way of representing the information that is captured and processed by the human visual system. Light fields can be acquired using compact imaging systems and the information can be used for a variety of applications including analysis, interpretation and display. Light field display architectures enable true 3D displays that provide a natural and comfortable experience. In both imaging and display, the system design has to consider a complex set of tradeoffs. This seminar will provide an introduction to light fields, followed by an in-depth discussion of state-of-the-art light field imaging and display systems, some interesting applications such as AR and VR and the key tradeoffs that have to be considered during the design process.

In this paper, we propose a design utilizing two identically parallel-aligned nematic liquid crystal (LC) plates for fast-response and polarization-independent phase modulator. Driven by synchronized voltage signals, such a polarizer-free variable phase modulator shows a wide tunable range from zero to more than 3π, back and forth at 532nm. Due to the optical compensation of the two plates, the rise and fall time of the phase retardation corresponds to the switching-on time of the two plates. Several advantages are illustrated based on the optical compensation of two identical parallel-aligned plates. First, zero phase retardation is obtained, which overcomes the residual phase due to surfaced anchored liquid crystal molecules. The second advantage is sub-millisecond response of rise and fall of retardation since simultaneous relaxation of the two plates remains optically hidden during the synchronized voltages fall. This fast-response and polarization-independent phase modulator has great potential for practical use, including optical communications and light field imaging systems.

The light pen is one of the most useful spatial coordinate measurement methods. However, due to the principle of vision measurement, the traditional light pen has the disadvantages such as limited measurement range, large structure size and complex calibration process. In order to revolutionize the design of light pen, this paper proposes a full space coordinate measurement method with a portable light pen by using hybrid light field imaging. Firstly, a light field imaging system with depth estimation ability has been introduced to simplify the light pen structure to three target points by using P3P method. In order to improve the accuracy of the depth estimation, a hybrid imaging method and EPI method have been used. Secondly, a full spatial measurement method with a rotation platform has been presented in detail. Finally, the experiments have been designed to verify the effectiveness and accuracy of the proposed method.

Light-field (LF) camera holds great promise for passive/general depth estimation benefited from high angular resolution, yet suffering small baseline for distanced region. While stereo solution with large baseline is superior to handle distant scenarios, the problem of limited angular resolution becomes bothering for near objects. Aiming for all-in-depth solution, we propose a cross-baseline LF camera using a commercial LF camera and a monocular camera, which naturally form a 'stereo camera' enabling compensated baseline for LF camera. The idea is simple yet non-trivial, due to the significant angular resolution gap and baseline gap between LF and stereo cameras.

A method is proposed for depth extraction of low-texture region with a light field (LF) camera, which expands the application. Based on the analysis of LF data, it is proven that the depth information can be estimated from a single LF image. Furthermore, as the lenslet LF data can be decoded into a subimages array, and the relationship among subimages is proven to be affine transformation, we used the geometry relationship, which is represented by partition ratio of triangle grids area, to replace the unreliable gray value of low-texture region for stereo matching. In addition, to obtain accurate ratio values, preset points are projected to enrich the texture with a projector, which is convenient and reliable. The proposed method improves the accuracy of the depth extraction obviously at low-texture region compared with the traditional state-of-the-art LF method, and results are validated by experiments.

<p>Light field photography offers a new approach to digitally captured images. These commercially available cameras are able to capture the 4D light field in a single image. This allows for a variety of image processing capabilities that traditional cameras do not offer. For example, the image can be digitally refocused after it is captured and its depth can be estimated. In terms of application, these capabilities could be beneficial on airborne platforms. However, a limitation of currently available light field cameras is that they are not fully functional at medium or long ranges. If these cameras were to capture light fields at longer ranges, they would have a practical application when mounted on low-flying aircrafts. This dissertation takes current light field photography techniques and modifies them so they work better to capture medium-range images. The majority of cameras that capture the 4D light field use a microlens array to modulate the incoming light before it hits the image sensor. Previous work using printed modulation masks garnered the same effect obtained by microlens arrays. This dissertation details the development of a modulation mask that has medium-range applications. A new way of extracting the 4D light field from raw images that uses a digital Fourier transform is presented. This method works for images captured with microlens arrays and printed mask cameras. Two prototype cameras were built and tested to demonstrate some of these concepts. The concepts demonstrated by these cameras could be used in the future designs of light field cameras.</p>

With the availability of commercial Light Field (LF) cameras, LF imaging has emerged as an up-and-coming technology in computational photography. However, the spatial resolution is significantly constrained in commercial micro-lens-based LF cameras because of the inherent multiplexing of spatial and angular information. Therefore, it becomes the main bottleneck for other applications of light field cameras. This paper proposes an adaptation module in a pre-trained Single Image Super-Resolution (SISR) network to leverage the powerful SISR model instead of using highly engineered light field imaging domain-specific Super Resolution models. The adaption module consists of a Sub-aperture Shift block and a fusion block. It is an adaptation in the SISR network to further exploit the spatial and angular information in LF images to improve the super-resolution performance. Experimental validation shows that the proposed method outperforms existing light field super-resolution algorithms. It also achieves PSNR gains of more than 1 dB across all the datasets as compared to the same pre-trained SISR models for scale factor 2, and PSNR gains 0.6 − 1 dB for scale factor 4.

Recently, light field cameras have drawn much attraction for their innovative performance in photographic and scientific applications. However, narrow baselines and constrained spatial resolution of current light field cameras impose restrictions on their usability. Therefore, we design a hybrid imaging system containing a light field camera and a high-resolution digital single lens reflex camera, and these two kinds of cameras share the same optical path with a beam splitter so as to achieve the reconstruction of high-resolution light fields. The high-resolution 4D light fields are reconstructed with a phase-based perspective variation strategy. First, we apply complex steerable pyramid decomposition on the high-resolution image from the digital single lens reflex camera. Then, we perform phase-based perspective-shift processing with the disparity value, which is extracted from the upsampled light field depth map, to create high-resolution synthetic light field images. High-resolution digital refocused images and high-resolution depth maps can be generated in this way. Furthermore, controlling the magnitude of the perspective shift enables us to change the depth of field rendering in the digital refocused images. We show several experimental results to demonstrate the effectiveness of our approach.

Light Field, a computational photography technique has various exciting features like refocusing and novel view synthesis. Previous works in light field photography have focused their work on creating light field images having a limited field of view. This limitation of light field images motivates the research for a acquisition system which can capture light field images with larger field of view. Recent works have been done to acquire light fields using a cylindrical arrangement of cameras which resolve the conventional complexity issue of acquisition of light field and subsequently merging them by essentially combining the capture and merging processes into a single exercise. We use this cylindrical system to generate a novel True Zoomed Light field panorama from the captured light field, which gives a wide 360? field of view similar to traditional panoramas along with the zoom applied to the whole panorama. The captured panorama can also be manipulated in various ways as is possible with other light field recordings. Also various zooming options which can be easily found in traditional camera, either tends to change the relative shape of the objects in image or introduces pix-elation effect. We use true zoom so as to improvise the zooming effect. We generate novel “True Zoomed Light Field Panorama” using our system. This type of panorama gives user a feeling of being closer to or farther from the whole environment at the same time.

This demonstration shows a 4-Degree-of-Freedom (4-DoF) parallax tolerant light field (LF) stitching system to generate large field of view (FoV) LF with interactive LF experiences. It can capture the LFs in real-time, generate the large FoV LF with high visual quality and visualize the advantage of large FoV LF by interactive 3D applications like 4-DoF stereo panorama, digital refocusing and 3D point cloud reconstruction.

Lens arrays enable us to easily achieve inexpensive light field acquisition in comparison with massive camera arrays. However, their micro-lenses often cause insufficient exposure to obtain high quality multi-view videos like camera arrays. In this paper, we propose a novel virtual camera array acquiring dynamic light fields as multi-view videos robustly and inexpensively, where mirror arrays consisting of tangent planes to a paraboloid are effectively combined with a single video camera. First, we demonstrate multi-view imaging of not an object but ordinary scenes with sufficient exposure by using a simple horizontal mirror array implemented as a prototype. Next, we theoretically reveal that there exists smooth extension of our mirror array to the vertical direction, that can be implemented easily by 3D printing, CNC machines, and so on. Finally, our design of light field acquisition is improved for effectively utilizing the whole of its mirror array by introducing an one-way mirror to prevent the camera itself from appearing on multi-view images.