Diffraction-Assisted Light Field Microscopy for Microtomography and Digital Volume Correlation with Improved Spatial Resolution

Light field technique at a single shot can get the whole volume image of observed sample. Therefore, the original frame rate of the optical system can be taken as the volumetric image rate. For dynamically imaging whole micron-scale biosample, a light field microscope with temporal focusing illumination has been developed. In the light field microscope, the f-number of the microlens array (MLA) is adopted to match that of the objective; hence, the subimages via adjacent lenslets do not overlay each other. A three-dimensional (3D) deconvolution algorithm is utilized to deblur the out-of-focusing part. Conventional light field microscopy (LFM) illuminates whole volume sample even noninteresting parts; nevertheless, whole volume excitation causes even more damage on bio-sample and also increase the background noise from the out of range. Therefore, temporal focusing is integrated into the light field microscope for selecting the illumination volume. Herein, a slit on the back focal plane of the objective is utilized to control the axial excitation confinement for selecting the illumination volume. As a result, the developed light field microscope with the temporal focusing multiphoton illumination (TFMPI) can reconstruct 3D images within the selected volume, and the lateral resolution approaches to the theoretical value. Furthermore, the 3D Brownian motion of two-micron fluorescent beads is observed as the criterion of dynamic sample. With superior signal-to-noise ratio and less damage to tissue, the microscope is potential to provide volumetric imaging for vivo sample.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Light field (LF) image super-resolution (SR) aims at reconstructing high-resolution LF images from their low-resolution counterparts. Although CNN-based methods have achieved remarkable performance in LF image SR, these methods cannot fully model the non-local properties of the 4D LF data. In this paper, we propose a simple but effective Transformer-based method for LF image SR. In our method, an angular Transformer is designed to incorporate complementary information among different views, and a spatial Transformer is developed to capture both local and long-range dependencies within each sub-aperture image. With the proposed angular and spatial Transformers, the beneficial information in an LF can be fully exploited and the SR performance is boosted. We validate the effectiveness of our angular and spatial Transformers through extensive ablation studies, and compare our method to recent state-of-the-art methods on five public LF datasets. Our method achieves superior SR performance with a small model size and low computational cost. Code is available at https://github.com/ZhengyuLiang24/LFT.

As light field images continue to increase in use and application, it becomes necessary to adapt existing image processing methods to this unique form of photography. In this paper we explore methods for applying neural style transfer to light field images. Feed-forward style transfer networks provide fast, high-quality results for monocular images, but no such networks exist for full light field images. Because of the size of these images, current light field data sets are small and are insufficient for training purely feed-forward style-transfer networks from scratch. Thus, it is necessary to adapt existing monocular style transfer networks in a way that allows for the stylization of each view of the light field while maintaining visual consistencies between views. To do this, we first generate disparity maps for each view given a single depth image for the light field. Then in a fashion similar to neural stylization of stereo images, we use disparity maps to enforce a consistency loss between views and to warp feature maps during the feed forward stylization. Unlike previous work, however, light fields have too many views to train a purely feed-forward network that can stylize the entire light field with angular consistency. Instead, the proposed method uses an iterative optimization for each view of a single light field image that backpropagates the consistency loss through the network. Thus, the network architecture allows for the incorporation of pre-trained fast monocular stylization network while avoiding the need for a large light field training set.

Light field (LF) image super-resolution (SR) can improve the limited spatial resolution of LF images by using complementary information from different perspectives. However, current LF image SR methods only use the RGB data to implicitly exploit the information among different perspectives, without paying attention to the information loss from raw data to RGB data and the explicit structure information utilization. To address the first issue, a data generation pipeline is developed to collect LF raw data for LF image SR. In addition, to make full use of the multiview information, an end-to-end convolutional neural network architecture (namely, LF-RawSR) is proposed for LF image SR. Specifically, an aggregated module is first used to fuse the angular information based on a volume transformer with plane sweep volume. Then the aggregated feature is warped to all LF views using a cross-view transformer for nonlocal dependencies utilization. The experimental results demonstrate that our method outperforms existing state-of-the-art methods with a comparative computational cost, and fine details and clear structures can be restored.

In this paper, a novel microscopic imaging technology, termed light field Fourier ptychographic microscopy (LFFPM), is presented which enables the reconstruction of wide-field, high-resolution images with the depth information by utilizing a light-emitting-diode (LED) illumination matrix.Conventional light field microscope lacks the ability of high lateral resolution because it sacrifices its lateral resolution to its angular resolution. Compared to conventional light field microscope, LFFPM enhances the lateral resolution with no need for the object to be in focus. Using LFFPM, the result shows that the lateral resolution is more than 2 times better than light field microscopy in simulation and 1.5 times in the experiment. Furthermore, the simulation result also shows that the ability of LFFPM to recovery the information of the phase and amplitude.Thus, this technology has a wide prospect of various biomedical applications such as neuroanatomy and haematology.

Light Field, a computational photography technique has various exciting features like refocusing and novel view synthesis. Previous works in light field photography have focused their work on creating light field images having a limited field of view. This limitation of light field images motivates the research for a acquisition system which can capture light field images with larger field of view. Recent works have been done to acquire light fields using a cylindrical arrangement of cameras which resolve the conventional complexity issue of acquisition of light field and subsequently merging them by essentially combining the capture and merging processes into a single exercise. We use this cylindrical system to generate a novel True Zoomed Light field panorama from the captured light field, which gives a wide 360? field of view similar to traditional panoramas along with the zoom applied to the whole panorama. The captured panorama can also be manipulated in various ways as is possible with other light field recordings. Also various zooming options which can be easily found in traditional camera, either tends to change the relative shape of the objects in image or introduces pix-elation effect. We use true zoom so as to improvise the zooming effect. We generate novel “True Zoomed Light Field Panorama” using our system. This type of panorama gives user a feeling of being closer to or farther from the whole environment at the same time.

The light field image, also known as the plenoptic image, contains the information about not only the intensity of light in a scene but also the direction of the light rays in space. Since the light field image contains very rich photometric and geometric information, it will have very widespread application in the future. For example, immersive content capture for virtual and mixed reality presentation or depth from light field for auto driving applications. To be more specific, a light field image can be enhanced with physical models for an autonomous decision making process, which is also an important task of the Dynamic Data Driven Applications Systems (DDDAS) [11]. Besides, the rich geometry and photometric information contained in the light field can be updated with real-time measurements, which is a focus of DDDAS such as smart city related image and video processing tasks. However, to make the light field images easier to be utilized, one of the most important tasks is to compress the light field images efficiently so they can be easily distributed over the current communication infrastructure.

In the past few years the integral-imaging, or lightfield, concept has been applied successfully to microscopy. More specifically, in the case of fluorescent samples integral, or lightfield, microscopy offers the advantage of capturing the 3D information in a single shot. Due to its potential utility integral microscopy is now facing many challenges, like improving the resolution and depth of field, the development and optimization of specially-adapted reconstruction algorithms, or the search of applications in which lightfield microscopy is superior to existing techniques. This contribution is devoted to review the recent advances of integral microscopy and enunciate the right questions about the progress of the technique.

Light field images can record multiple information about the light rays in a scene and provide multiple views from a single image, offering a new data source for 3D reconstruction. However, ensuring the quality of light field images themselves is challenging, and distorted image inputs may lead to poor reconstruction results. Accurate light field image quality assessment can pre-judge the quality of light field images used as input for 3D reconstruction, providing a reference for the reconstruction results before the reconstruction work, significantly improving the efficiency of 3D reconstruction based on light field images. In this paper, we propose an Adaptive Vision Transformer-based light-field image-quality assessment model (AViT-LFIQA). The model adopts a multi-view sub-aperture image sequence input method, greatly reducing the number of input images while retaining as much information as possible from the original light field image, alleviating the training pressure on the neural network. Furthermore, we design an adaptive learnable attention layer based on ViT, which addresses the lack of inductive bias in ViT by using adaptive diagonal masking and a learnable temperature coefficient strategy, making the model more suitable for training on small datasets of light field images. Experimental results demonstrate that the proposed model is effective for various types of distortions and shows superior performance in light-field image-quality assessment.

Light field imaging is an emerging technology in computational photography areas. Based on innovative designs of the imaging model and the optical path, light field cameras not only record the spatial intensity of threedimensional (3D) objects, but also capture the angular information of the physical world, which provides new ways to address various problems in computer vision, such as 3D reconstruction, saliency detection, and object recognition. In this paper, three key aspects of light field cameras, i.e., model, calibration, and reconstruction, are reviewed extensively. Furthermore, light field based applications on informatics, physics, medicine, and biology are exhibited. Finally, open issues in light field imaging and long-term application prospects in other natural sciences are discussed.

3D localization of point source is widely used in many fields, such as bioimaging and autonomous driving fields. However, the localization is hard to perform under scattering conditions because of the diffuse effect of the scattering. We propose a novel method for 3D localization of point source under scattering conditions based on light field imaging and deep learning by only one shot. First, we introduce the description of the point source in a light field wise and how to localize a point source by its light field. On the basis, we elaborate on the effect of scattering on a light field and how to retrieve the location of a point source from a light field with scattering. Then, the effect of aberration on a light field will be introduced. We also build an artificial scene and a deep learning framework to perform a 3D localization practically, and the feasibility and accuracy of our method have been evaluated.

Recent advances in light field technologies is fostering the research and development of novel imaging applications. Such applications perform processing of the light field information to create new visual effects such as, for example, refocusing, perspective change, colour adjustment. Light field imaging is very data intensive compared with usual digital photographic imaging, and novel compression algorithms are needed for addressing the problem of light field storage and transmission. Raw digital light field images exhibit low spatial correlation if compared to regular images and hence the performance in terms of rate-distortion of current state-of-the-art image encoders can be superseded by devising novel image compression architectures. In this paper, an architecture for lossy compression of unfocused light field images is proposed. Raw light fields are preprocessed by demosaicing, devignetting and slicing of the raw lenslet array image. The slices are then compressed with the JPEG 2000 image coding standard. The performance of the proposed method is compared against direct application of JPEG 2000 compression on the 4D light field. The experimental analysis has been conducted under a set of different compression ratios and the obtained results show that the proposed method outperforms direct application of the reference architecture.

Ear recognition is an emerging research area in image-based biometrics. The commercial availability of lenslet light field cameras able to capture full spatio-angular information has brought momentum to biometric and forensic research exploiting this new type of imaging sensors. This study is the first to consider the usage of light field cameras for ear recognition, proposing both an appropriate content database and an ear recognition solution. The novel Lenslet Light Field Ear DataBase (LLFEDB) includes 536 light field images corresponding to four different poses, from 67 subjects, captured with a Lytro ILLUM lenslet light field camera. The LLFEDB includes critical cases such as ear images partly occluded by ear piercing, earing, hair and combination of occlusions. The novel light field-based ear recognition solution proposed exploits the richer spatio-angular information available in the light field images. A comparative performance evaluation study using the LLFEDB, and focusing on the most recent light field and non-light field based descriptors for ear recognition, shows a very promising performance of the proposed descriptor, outperforming all the assessed descriptors.

Existing light field (LF) stitching approaches mainly consider the geometric correspondence and ignore the high-level semantic information, so it is difficult to ensure the angular consistency of the stitched results while aligning the LF images. To address this problem, this paper proposes a panoramic LF - oriented image stitching method with semantics and projection transformation. Specifically, semantic information is first extracted from the LF images and applied to classify feature points. Hence, all the feature points are endowed corresponding semantic information based on the classification result. Afterwards, the geometric and semantic information of the feature points are combined to estimate the local homography matrix to align the input LF images. Finally, a weighted fusion strategy is employed to fuse the aligned results. Experimental results show that the proposed method can not only guarantee the spatial quality of the stitched LF image. But also effectively preserve accurate angular consistency. Moreover, the proposed method outperforms the existing stitching methods in both subjective and objective comparisons.