Fast Physically Correct Refocusing for Sparse Light Fields Using Block-Based Multi-Rate View Interpolation.

This demonstration shows a 4-Degree-of-Freedom (4-DoF) parallax tolerant light field (LF) stitching system to generate large field of view (FoV) LF with interactive LF experiences. It can capture the LFs in real-time, generate the large FoV LF with high visual quality and visualize the advantage of large FoV LF by interactive 3D applications like 4-DoF stereo panorama, digital refocusing and 3D point cloud reconstruction.

Light field cameras have a wide area of applications, such as digital refocusing, scene depth information extraction and 3-D image reconstruction. By recording the energy and direction information of light field, they can well solve many technical problems that cannot be done by conventional cameras. An important feature of light field cameras is that a microlens array is inserted between the sensor and main lens, through which a series of sub-aperture images of different perspectives are formed. Based on this feature and the full-focus image acquisition technique, we propose a light-field optical flow calculation algorithm, which involves both the depth estimation and the occlusion detection and guarantees the edge-preserving property. This algorithm consists of three steps: 1) Computing the dense optical flow field among a group of sub-aperture images; 2) Obtaining a robust depth-estimation by initializing the light-filed optical flow using the linear regression approach and detecting occluded areas using the consistency; 3) Computing an improved light-field depth map by using the edge-preserving algorithm to realize interpolation optimization. The reliability and high accuracy of the proposed approach is validated by experimental results.

Recently, many image-based modeling and rendering techniques have been successfully designed to render photo-realistic images without the need for explicit 3D geometry. However, these techniques (e.g., light field rendering (Levoy, M. and Hanrahan, P., 1996. In SIGGRAPH 1996 Conference Proceedings, Annual Conference Series, Aug. 1996, pp. 31–42) and Lumigraph (Gortler, S.J., Grzeszczuk, R., Szeliski, R., and Cohen, M.F., 1996. In SIGGRAPH 1996 Conference Proceedings, Annual Conference Series, Aug. 1996, pp. 43–54)) may require a substantial number of images. In this paper, we adopt a geometric approach to investigate the minimum sampling problem for light field rendering, with and without geometry information of the scene. Our key observation is that anti-aliased light field rendering is equivalent to eliminating the “double image” artifacts caused by view interpolation. Specifically, we present a closed-form solution of the minimum sampling rate for light field rendering. The minimum sampling rate is determined by the resolution of the camera and the depth variation of the scene. This rate is ensured if the optimal constant depth for rendering is chosen as the harmonic mean of the maximum and minimum depths of the scene. Moreover, we construct the minimum sampling curve in the joint geometry and image space, with the consideration of depth discontinuity. The minimum sampling curve quantitatively indicates how reduced geometry information can be compensated by increasing the number of images, and vice versa. Experimental results demonstrate the effectiveness of our theoretical analysis.

A new multiview just-noticeable-depth-difference(MJNDD) Model is presented and applied to compress the joint multiview video plus depth. Many video coding algorithms remove spatial and temporal redundancies and statistical redundancies but they are not capable of removing the perceptual redundancies. Since the final receptor of video is the human eyes, we can remove the perception redundancy to gain higher compression efficiency according to the properties of human visual system (HVS). Traditional just-noticeable-distortion (JND) model in pixel domain contains luminance contrast and spatial-temporal masking effects, which describes the perception redundancy quantitatively. Whereas HVS is very sensitive to depth information, a new multiview-just-noticeable-depth-difference(MJNDD) model is proposed by combining traditional JND model with just-noticeable-depth-difference (JNDD) model. The texture video is divided into background and foreground areas using depth information. Then different JND threshold values are assigned to these two parts. Later the MJNDD model is utilized to encode the texture video on JMVC. When encoding the depth video, JNDD model is applied to remove the block artifacts and protect the edges. Then we use VSRS3.5 (View Synthesis Reference Software) to generate the intermediate views. Experimental results show that our model can endure more noise and the compression efficiency is improved by 25.29 percent at average and by 54.06 percent at most compared to JMVC while maintaining the subject quality. Hence it can gain high compress ratio and low bit rate.

In this paper, we use a pinhole-array-mask light field camera to generate duplicated spectra in the frequency domain. Based on Fourier-Slicing Theorem, we can collect light field information on different image planes by taking different data-slices in the frequency domain. When a series of sliced data is converted back to the spatial domain, digital-refocusing can be performed. Since the digital refocusing process is time-consuming, we design a special-purpose processor for acceleration. The chip is synthesized with TSMC 90nm technology. A 15X speedup can be achieved when compared to the software version.

High-resolution and wide-range depth maps are the key to enable novel light-field applications, such as digital refocusing, view synthesis, and 3D reconstruction. In this paper, we present an energy-efficient depth-estimation processor on FPGA to meet this purpose. There are two major contributions. First, image-guided depth inference and upsampling is adopted and implemented to provide accurate depth maps while lowering the working frequency from 215MHz to 54MHz. Second, octave search range sampling is proposed to efficiently allocate depth labels for wide-depth-range scenes to save computation and maintain accuracy. Finally, the implementation result on Xilinx ZC706 shows ASIC-comparable energy efficiency—95pJ/label—for Full-HD five-view light fields at 30fps.

Light-field (LF) cameras [1] have significantly advanced imaging and microscopy by capturing richer multi-dimensional information than conventional 2D images, enabling applications such as depth estimation and digital refocusing. However, the inherent trade-off between spatial and angular resolution remains a key limitation. LF capture both spatial and angular information under a narrow field of view. Under ideal conditions, disparity propagation from the central view allows for warping to generate other sub-aperture views. However, occlusions and view-dependent effects often hinder this propagation.

Light field rendering (LFR) is an image-based method for generating novel views from a set of camera images. When sample of camera array is sparse, conventional light field rendering would cause aliasing artifacts. We present a straightforward layer-based method that significantly removes the aliasing artifacts with no prior knowledge of the scene geometry. First, a sparse disparity map is generated to detect the depth layers of the scene. Then, for synthesized each pixel, using the color consistency constraint to detect the optimum focal plane from the layer depths, and rendering such pixel with selected focal plane by LFR method. Finally, a trick method is used to reduce flickering artifacts caused by the color consistency constraint. Experimental results show the effectiveness of our approach.

With the development of consumer light field cameras, the light field imaging has become an extensively used method for capturing the three-dimensional appearance of a scene. The depth estimation often requires a dense sampled light field in the angular domain or a high resolution in the spatial domain. However, there is an inherent trade-off between the angular and spatial resolutions of the light field. Recently, some studies for super-resolving the trade-off light field have been introduced. Rather than the conventional approaches that optimize the depth maps, these approaches focus on maximizing the quality of the super-resolved light field. In this article, we investigate how the depth estimation can benefit from these super-resolution methods. Specifically, we compare the qualities of the estimated depth using (a) the original sparse sampled light fields and the reconstructed dense sampled light fields, and (b) the original low-resolution light fields and the high-resolution light fields. Experiment results evaluate the enhanced depth maps using different super-resolution approaches.

4D Light Field (LF) imaging, since it conveys both spatial and angular scene information, can facilitate computer vision tasks and generate immersive experiences for end-users. A key challenge in 4D LF imaging is to flexibly and adaptively represent the included spatio-angular information to facilitate subsequent computer vision applications. Recently, image over-segmentation into homogenous regions with perceptually meaningful information has been exploited to represent 4D LFs. However, existing methods assume densely sampled LFs and do not adequately deal with sparse LFs with large occlusions. Furthermore, the spatio-angular LF cues are not fully exploited in the existing methods. In this paper, the concept of hyperpixels is defined and a flexible, automatic, and adaptive representation for both dense and sparse 4D LFs is proposed. Initially, disparity maps are estimated for all views to enhance over-segmentation accuracy and consistency. Afterwards, a modified weighted K -means clustering using robust spatio-angular features is performed in 4D Euclidean space. Experimental results on several dense and sparse 4D LF datasets show competitive and outperforming performance in terms of over-segmentation accuracy, shape regularity and view consistency against state-of-the-art methods.

The autonomous underwater vehicle (AUV) frequently operates in harsh underwater environments, and timely fault diagnosis of the AUV can prevent mission failure and equipment loss. Data‐driven methods based on a single data source have been widely utilized for fault diagnosis of the AUV because they do not require the construction of complex mechanism models and have high fault diagnosis accuracy. However, these methods face challenges in accomplishing complex fault diagnosis tasks because the single data source provides very restricted fault features. To address this issue, an attention mechanism‐based multisensor data fusion neural network (MDFNN) for AUV fault diagnosis is proposed in this work. First, a feature extraction layer based on the two‐dimensional (2D) convolutional method with a 1D kernel is introduced to extract features from each sensor data separately, significantly optimizing the model architecture. Second, an efficient channel attention mechanism‐based feature fusion layer is proposed to reassign weights to the features of each sensor data, enabling the model to focus more on crucial features. Finally, the fused features are input to the fully connected layers and softmax layer to realize the fault diagnosis of multisensor data. In the end, the diagnostic performance of the proposed MDFNN is evaluated utilizing real AUV experimental data. The experiment shows that the proposed MDFNN has a very fast convergence speed and 98.37% fault diagnosis accuracy, demonstrating its excellent fault diagnosis performance. The proposed MDFNN provides a generalized and simply structured fault diagnosis framework for the AUV with multiple types of sensor data, providing significant engineering value.

This thesis covers a broad part of the field of computational photography, including video stabilization and image warping techniques, introductions to light field photography and the conversion of monocular images and videos into stereoscopic 3D content. We present a user assisted technique for stereoscopic 3D conversion from 2D images. Our approach exploits the geometric structure of perspective images including vanishing points. We allow a user to indicate lines, planes, and vanishing points in the input image, and directly employ these as guides of an image warp that produces a stereo image pair. Our method is most suitable for scenes with large scale structures such as buildings and is able to skip the step of constructing a depth map. Further, we propose a method to acquire 3D light fields using a hand-held camera, and describe several computational photography applications facilitated by our approach. As the input we take an image sequence from a camera translating along an approximately linear path with limited camera rotations. Users can acquire such data easily in a few seconds by moving a hand-held camera. We convert the input into a regularly sampled 3D light field by resampling and aligning them in the spatio-temporal domain. We also present a novel technique for high-quality disparity estimation from light fields. Finally, we show applications including digital refocusing and synthetic aperture blur, foreground removal, selective colorization, and others.

In this paper, we present a novel postprocessing technique based on the theory of the projection onto convex sets (POCS) in order to reduce the blocking artifacts in digital high definition television (HDTV) images. By detecting and eliminating the undesired high-frequency components, mainly caused by blocking artifacts, we propose a new smoothness constraint set (SCS) and its projection operator in the DCT domain. In addition, we propose an improved quantization constraint set (QCS) using the correlation of DCT coefficients between adjacent blocks. In the proposed technique, the range of the QCS is efficiently reduced as close to the original DCT coefficient as possible to yield better performance of the projection onto the QCS. Computer simulation results indicate that the proposed schemes perform better than conventional algorithms. Furthermore, we introduce a fast implementation method of the proposed algorithm. The conventional POCS-based postprocessing techniques require the forward/inverse discrete cosine transform (DCT/IDCT) operations with a heavy computational burden. To reduce the computational complexity we introduce a fast implementation method of the proposed algorithm that does not perform DCT/IDCT operations. Estimates of computation savings vary between 41% and 64% depending on the task.

Computing the light field due to an area light source remains an interesting problem in computer graphics. The paper presents a series approximation of the light field due to an unoccluded area source, by expanding the light field in spherical harmonics. The source can be nonuniform and need not be a planar polygon. The resulting formulas give expressions whose cost and accuracy can be chosen between the exact and expensive Lambertian solution for a diffuse polygon, and the fast but inexact method of replacing the area source by a point source of equal power. The formulas break the computation of the light vector into two phases: the first phase represents the light source's shape and brightness with numerical coefficients, and the second uses these coefficients to compute the light field at arbitrary locations. The author examines the accuracy of the formulas for spherical and rectangular Lambertian sources, and applies them to obtain light gradients. The author also shows how to use the formulas to estimate light from uniform polygonal sources, sources with polynomially varying radiosity, and luminous textures.

Speckle tracking method has been studied for velocity vector imaging in medical ultrasound for many years. However, the method is slow as compared to the conventional color Doppler imaging due to a larger amount of computation is needed. In this study, a method is developed to reduce the amount of computation. Instead of using an entire rectangular block in a 2D kernel (full kernel) to calculate the sum of absolute difference between image frames, only data in two diagonal lines, two bisectors, or a combination of the diagonal and bisector lines (see Fig. 1(b), (c), and (d)), are used. Both simulation and experiment were conducted to verify the proposed method. 11 frames of images are reconstructed with the delay-and-sum (D&S) and the high-frame-rate (HFR) imaging methods of a single plane wave transmission. In the simulation, two point scatterers, located at depths of 10 and 70 mm with velocities of 0.5 and 1.0 m/s respectively, are placed in the imaging area. Ten velocities are estimated by speckle tracking using both the full kernel and the crosses. The average errors of velocities measured are 7.3% for D&S and 5.5% for HFR imaging methods with diagonal lines, 2.5% for D&S and 4.0% for HFR imaging methods with bisectors, 3.6% for D&S and 3.2% for HFR imaging methods when both diagonal and bisector lines are used, and 2.6% for D&S and 2.5% for HFR imaging methods based on the full kernel. In the experiment, two small glass beads (point scatterers), located at depths of around 10 and 70 mm with velocities of 0 and a fixed value respectively, were placed in a water tank. The average measurement errors are 15.5% for D&S and 11.4% for HFR imaging methods with diagonal lines, 10.7% for D&S and 12.5% for HFR imaging methods with bisectors, 13.2% for D&S and 9.3% for HFR imaging methods when both diagonal and bisector lines are used, and 10.3% for D&S and 8.7% for HFR imaging methods with the full kernel. These results were obtained with a kernel area of 1.8 × 1.8 mm (or 100 × 100=10000 pixels for the full kernel and 4 × 100=400 pixels for a combination of diagonal and bisector lines). In this condition, the computation time is 18.5 times smaller with the combined diagonal and bisector lines than that with the full kernel (excluding the fixed image reading time). As the size of the kernel is increased, there will be more reduction in computation time with the new method. The new speckle tracking method can achieve similar accuracies in velocity estimations while significantly reducing the computation time.