Underwater Imaging System Based on Light Field Technology

In this paper, an aberration correction method for an extended target is proposed to solve the problem of the lenslet-based plenoptic camera not imaging clearly under the influence of aberrations. We propose a light field manipulation method to improve performance of the light field imaging system. The principle of this method is that the sub-aperture images extracted from the raw light field image are offset when the light field imaging system is affected by aberrations, and the symmetrical arrangement of the sub-aperture image array is destroyed. By repairing the symmetrical arrangement of the sub-aperture image array, the influence of phase aberrations on the imaging system can be eliminated, and the resolution of the plenoptic camera can be improved. We use an image correlation algorithm to process the sub-aperture images of the plenoptic camera, calculate and compensate each sub-aperture image's displacement caused by aberrations, and restore the symmetrical arrangement of the sub-aperture image array; then, a corrected high-resolution refocused image can be generated. In particular, this method uses only the raw light field information obtained by the plenoptic camera in a single exposure, without adding other hardware devices. Furthermore, it takes the extended target itself as the reference image, so the ideal position need not be calibrated in advance. Also, the parallax information of the sub-aperture images is retained, and the method is simple and easy to use. Numerical simulation and experimental results show that the technology proposed in this paper can work well for high-resolution imaging of a plenoptic camera with phase aberrations. This method can be potentially applied to analyze lens aberration, media-induced image distortion such as water turbulence in underwater imaging, and atmospheric turbulence in remote imaging. It may have important application prospects in the fields of astronomical object detection, remote sensing, etc.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Recent advances in light field technologies is fostering the research and development of novel imaging applications. Such applications perform processing of the light field information to create new visual effects such as, for example, refocusing, perspective change, colour adjustment. Light field imaging is very data intensive compared with usual digital photographic imaging, and novel compression algorithms are needed for addressing the problem of light field storage and transmission. Raw digital light field images exhibit low spatial correlation if compared to regular images and hence the performance in terms of rate-distortion of current state-of-the-art image encoders can be superseded by devising novel image compression architectures. In this paper, an architecture for lossy compression of unfocused light field images is proposed. Raw light fields are preprocessed by demosaicing, devignetting and slicing of the raw lenslet array image. The slices are then compressed with the JPEG 2000 image coding standard. The performance of the proposed method is compared against direct application of JPEG 2000 compression on the 4D light field. The experimental analysis has been conducted under a set of different compression ratios and the obtained results show that the proposed method outperforms direct application of the reference architecture.

Light fields provide a natural way of representing the information that is captured and processed by the human visual system. Light fields can be acquired using compact imaging systems and the information can be used for a variety of applications including analysis, interpretation and display. Light field display architectures enable true 3D displays that provide a natural and comfortable experience. In both imaging and display, the system design has to consider a complex set of tradeoffs. This seminar will provide an introduction to light fields, followed by an in-depth discussion of state-of-the-art light field imaging and display systems, some interesting applications such as AR and VR and the key tradeoffs that have to be considered during the design process.

.Significance: Light-field fundus photography has the potential to be a new milestone in ophthalmology. Up-to-date publications show only unsatisfactory image quality, preventing the use of depth measurements. We show that good image quality and, consequently, reliable depth measurements are possible, and we investigate the current challenges of this novel technology.Aim: We investigated whether light field (LF) imaging of the retina provides depth information, on which structures the depth is estimated, which illumination wavelength should be used, whether deeper layers are measurable, and what kinds of artifacts occur.Approach: The technical setup, a mydriatic fundus camera with an LF imager, and depth estimation were validated by an eye model and in vivo measurements of three healthy subjects and three subjects with suspected glaucoma. Comparisons between subjects and the corresponding optical coherence tomography (OCT) measurements were used for verification of the depth estimation.Results: This LF setup allowed for three-dimensional one-shot imaging and depth estimation of the optic disc with green light. In addition, a linear relationship was found between the depth estimates of the OCT and those of the setup developed here. This result is supported by the eye model study. Deeper layers were not measurable.Conclusions: If image artifacts can be handled, LF technology has the potential to help diagnose and monitor glaucoma risk at an early stage through a rapid, cost-effective one-shot technology.

The light pen is one of the most useful spatial coordinate measurement methods. However, due to the principle of vision measurement, the traditional light pen has the disadvantages such as limited measurement range, large structure size and complex calibration process. In order to revolutionize the design of light pen, this paper proposes a full space coordinate measurement method with a portable light pen by using hybrid light field imaging. Firstly, a light field imaging system with depth estimation ability has been introduced to simplify the light pen structure to three target points by using P3P method. In order to improve the accuracy of the depth estimation, a hybrid imaging method and EPI method have been used. Secondly, a full spatial measurement method with a rotation platform has been presented in detail. Finally, the experiments have been designed to verify the effectiveness and accuracy of the proposed method.

Perceptual encryption has received widespread attention as a technology for protecting multimedia visual information. In multimedia applications, perceptual encryption only protects a portion of the content without previewing all the information. At present, perceptual encryption is mostly oriented to conventional plain images while few methods aim at visual security measures for perceptually encrypted images. Existing solutions usually adopt well-known quality assessment metrics to measure the visual quality of encrypted images. However, they often exhibit undesired behavior on perceptually encrypted images with low quality. As a typical representation of three-dimensional scenes, light field images record the intensity and direction of light during propagation which is distinct from 2-D images. In this paper, we construct a perceptually encrypted light field image database (PE-LFID) for quality assessment based on 14 reference light field plaintext images. For each scene, we employ four encryption methods, each of which has six levels. Additionally, a novel visual security evaluation method based on PE-LFID is proposed by taking into account the local and global features of the light field images. First, we use a multi-threshold edge detection method to obtain the edge similarity in the spatial domain of the light field image. Afterwards, the epipolar plane image (EPI) generated from the angular domain of the light field image is used to calculate the gradient magnitude similarity, which is expressed as a global feature. Furthermore, the final quality prediction score is calculated by adaptively weighting between local and global features. We conduct extensive experiments on the proposed PE-LFID to assess the performance of classical and state-of-the-art IQA models. The experimental results demonstrate the effectiveness of the proposed method for visual security evaluation of perceptually encrypted light field images, as well as the scalability of PE-LFID.

Light field images can record multiple information about the light rays in a scene and provide multiple views from a single image, offering a new data source for 3D reconstruction. However, ensuring the quality of light field images themselves is challenging, and distorted image inputs may lead to poor reconstruction results. Accurate light field image quality assessment can pre-judge the quality of light field images used as input for 3D reconstruction, providing a reference for the reconstruction results before the reconstruction work, significantly improving the efficiency of 3D reconstruction based on light field images. In this paper, we propose an Adaptive Vision Transformer-based light-field image-quality assessment model (AViT-LFIQA). The model adopts a multi-view sub-aperture image sequence input method, greatly reducing the number of input images while retaining as much information as possible from the original light field image, alleviating the training pressure on the neural network. Furthermore, we design an adaptive learnable attention layer based on ViT, which addresses the lack of inductive bias in ViT by using adaptive diagonal masking and a learnable temperature coefficient strategy, making the model more suitable for training on small datasets of light field images. Experimental results demonstrate that the proposed model is effective for various types of distortions and shows superior performance in light-field image-quality assessment.

Light field reconstruction from images captured by focal plane sweeping, such as light field moment imaging~(LFMI) and light field reconstruction with back projection~(LFBP), can achieve high transverse resolution comparable to the modern camera sensor. This is impossible for the conventional microlens-array-based light field imaging systems. However, the acquisition of a series of focal plane sweeping images along the optical axis is time consuming and requires fine alignment. Besides, different focal plane based light field reconstruction techniques require images with different characteristics. To solve these problems, we present an efficient approach for fast light field acquisition with precise focal plane sweeping capture by defocus modulation rather than mechanical movement. Because of the controllable point spread function, we can capture images for light field reconstruction with both LFMI and LFBP

Typical autonomous robotics systems incorporate multiple cameras, LIDAR sensors and sophisticated computing resources. In this paper we present a software framework for utilizing any array of multiple cameras with sufficient field-of-view (FOV) overlap as a light field imaging system. We show that the typical linear arrays that exist on autonomous cars are sufficient to capture stable time resolved light fields even when moving at highway speeds. We elaborate on the potential pitfalls associated with such a technique namely loss of calibration between cameras due to high frequency vibrations and sudden shocks associated with driving over potholes and highlight a method that can compensate for such effects. We demonstrate that the light fields collected by simple linear arrays can be processed in real time for a wide variety of useful applications including occlusion removal, for signal enhancement in featureless images captured in very low light, for reflection removal and for improved visibility in extreme conditions associated with snow and heavy rain.

Light field (LF) image super-resolution (SR) aims at reconstructing high-resolution LF images from their low-resolution counterparts. Although CNN-based methods have achieved remarkable performance in LF image SR, these methods cannot fully model the non-local properties of the 4D LF data. In this paper, we propose a simple but effective Transformer-based method for LF image SR. In our method, an angular Transformer is designed to incorporate complementary information among different views, and a spatial Transformer is developed to capture both local and long-range dependencies within each sub-aperture image. With the proposed angular and spatial Transformers, the beneficial information in an LF can be fully exploited and the SR performance is boosted. We validate the effectiveness of our angular and spatial Transformers through extensive ablation studies, and compare our method to recent state-of-the-art methods on five public LF datasets. Our method achieves superior SR performance with a small model size and low computational cost. Code is available at https://github.com/ZhengyuLiang24/LFT.

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

As light field images continue to increase in use and application, it becomes necessary to adapt existing image processing methods to this unique form of photography. In this paper we explore methods for applying neural style transfer to light field images. Feed-forward style transfer networks provide fast, high-quality results for monocular images, but no such networks exist for full light field images. Because of the size of these images, current light field data sets are small and are insufficient for training purely feed-forward style-transfer networks from scratch. Thus, it is necessary to adapt existing monocular style transfer networks in a way that allows for the stylization of each view of the light field while maintaining visual consistencies between views. To do this, we first generate disparity maps for each view given a single depth image for the light field. Then in a fashion similar to neural stylization of stereo images, we use disparity maps to enforce a consistency loss between views and to warp feature maps during the feed forward stylization. Unlike previous work, however, light fields have too many views to train a purely feed-forward network that can stylize the entire light field with angular consistency. Instead, the proposed method uses an iterative optimization for each view of a single light field image that backpropagates the consistency loss through the network. Thus, the network architecture allows for the incorporation of pre-trained fast monocular stylization network while avoiding the need for a large light field training set.

Light field (LF) image super-resolution (SR) can improve the limited spatial resolution of LF images by using complementary information from different perspectives. However, current LF image SR methods only use the RGB data to implicitly exploit the information among different perspectives, without paying attention to the information loss from raw data to RGB data and the explicit structure information utilization. To address the first issue, a data generation pipeline is developed to collect LF raw data for LF image SR. In addition, to make full use of the multiview information, an end-to-end convolutional neural network architecture (namely, LF-RawSR) is proposed for LF image SR. Specifically, an aggregated module is first used to fuse the angular information based on a volume transformer with plane sweep volume. Then the aggregated feature is warped to all LF views using a cross-view transformer for nonlocal dependencies utilization. The experimental results demonstrate that our method outperforms existing state-of-the-art methods with a comparative computational cost, and fine details and clear structures can be restored.

With the development of immersive media, Light Field Image (LFI) quality assessment is becoming more and more important, which helps to better guide light field acquisition, processing and application. However, almost all existing LFI quality assessment schemes utilize the 2D or 3D quality assessment methods while ignoring the intrinsic high dimensional characteristics of LFI. Therefore, we adopt the tensor theory to explore the LF 4D structure characteristics and propose the first Blind quality Evaluator of LIght Field image (BELIF). We generate cyclopean images tensor from the original LFI and then the features are extracted by the tucker decomposition. Specifically, Tensor Spatial Characteristic Features (TSCF) for spatial quality and Tensor Structure Variation Index (TSVI) for angular consistency are designed to fully assess the LFI quality. Extensive experimental results on the public LFI databases demonstrate that BELIF signifi-cantly outperforms the existing image quality assessment algorithms.