Real Time Monocular Depth from Defocus

A camera shake of the relative motion between cameras and original scenes often occurs during a shoot, resulting in blurred images. This paper presents a method of restoring blurred images based on patch-based PSF (Point Spread Function) estimation and channel-dependent image deblurring. Initially, a pair of noisy and blurred images is acquired by consecutive shooting under short and long exposures to more accurately estimate the 2D trajectory formed by the camera shake, which is also known as PSF. From the noisy image, one patch with strong edges is extracted using gradient operators and the corresponding blurry patch is found within the blurry image. The boundaries of both patches are padded in a specified tile pattern to prevent the appearance of vertical or horizontal lines due to boundary discontinuity. Tikhonov regularization is then iteratively conducted on Gaussian pyramid images with adaptive noise thresholding to get the optimized PSF. Finally, the color conversion of the blurred image into a luminance and two chrominance images is done to apply the channel dependent image deblurring where the spatial resolution of the chrominance images is reduced based on the natural gradient distribution to abandon the valueless data regardless of the restoration quality. Moreover, another artificially blurred image made by the convolution of the known Gaussian PSF and the noisy image is used to reduce undesirable ringing-artifacts and increase the sharpness. The experimental results show that the proposed method can provide more improved PSFs and restored images accompanied by the reduction of computing time and ring artifacts.

We propose a new method of blind deconvolution (BD) suitable for the recovery of truncated blurred images. In practical cases we often encounter truncated images, but they are difficult to treat properly with conventional BD algorithms, since nontruncated blurred images are assumed. The proposed method is based on simulated annealing (SA), to which we introduce an adaptive masking process. After a truncated segment is cut out from the blurred image, the sharp edges surrounding the segment are removed by multiplying the initial window, whose shape is determined with the initial estimate of the point spread function (PSF). A more appropriate estimate for the PSF is obtained from the masked segment based on the SA algorithm, and according to the new estimate of the PSF, the shape of the window is varied. These procedures are iterated until we finally obtain the best possible estimate of the PSF. In the last step, the entire blurred image is deconvolved with the final estimate of the PSF to obtain the recovered image. The effectiveness of the proposed method is confirmed by computer simulation.

A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. Over the last decade, a range of computational cameras have been proposed, which use various optics designs to encode and use computation to decode useful visual information. What is often missing, however, is the quantitative analysis of the relation between camera design and the captured visual information, and little systematic work has been done to evaluate and optimize these computational camera designs. While the optics of computational cameras may be quite complicated, many of them can be effectively characterized by their point spread functions (PSFs): the intensity distribution on an image sensor as a response to a point light source in a scene. This thesis explores the techniques to characterize, evaluate and optimize computational cameras via PSF engineering for computer vision tasks, including image recovery, 3D reconstruction, and image refocusing. I first explore PSF engineering techniques to recover image details from blurry images. Image blur is a problem commonly seen in photography due to a number of reasons, including defocus, lens aberration, diffraction, atmospheric turbulence, object motion, and etc. Image blur is often formulated as a convolution of the latent blur-free image and a PSF, and deconvolution (or deblurring) techniques have to be used to recover image details from a blurry region. Here, I propose a comprehensive framework of PSF evaluation for the purpose of image deblurring, in which the effects of image noise, deblurring algorithm, and the structure of natural images are all taken into account. In defocus blur, the shape of defocus PSF is largely determined by the aperture pattern of the camera lens. By using an evaluation criterion derived from the comprehensive framework, I optimize the aperture pattern to preserve many more image details when defocus occurs. Both through simulations and experiments, I demonstrate the significant improvement gained by using optimized aperture patterns. While defocus blur causes a loss in image detail, it also encodes depth information of scenes in images. A typical depth from defocus (DFD) technique computes depth from two or more images that are captured with circular aperture lenses of different focus settings. Circular apertures produce circular defocus PSFs. In this thesis, I show that the use of a circular aperture severely restricts the accuracy of DFD, and propose a comprehensive framework of PSF evaluation for depth recovery. With this framework, we can derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using a genetic algorithm and gradient descent search to arrive at a pair of high resolution apertures. The two coded apertures are found to complement each other in the scene frequencies they preserve. With this property it becomes possible to not only recover depth with greater fidelity but also to obtain a high quality all-focused image from the two defocused images. While depth recovery can significantly benefit from optimized aperture patterns, its overall performance is rigidly limited by the lens aperture's physical size. To transcend this limitation, I propose a novel depth recovery technique using an optical diffuser - referred to as depth from diffusion (DFDiff). I show that DFDiff is analogous to conventional DFD, in which the scatter angle of the diffuser determines the system's effective aperture. High precision depth estimation can be achieved by choosing a proper diffuser and no longer requires the large lenses that DFD requires. Even a consumer camera with a low-end small lens can be used to do high-precision depth estimation when coupled with an optical diffuser. In my detailed analysis of the image formation properties of a DFDiff system, I show a number of examples demonstrating greater precision in depth estimation when using DFDiff. While the finite depth of field (DOF) of a lens camera leads to defocus blur, it also produces artistic visual experience. Many of today's displays are interactive in nature, which opens up a possibility for new kind of visual representations. Users could, for example, interactively refocus images to different depths, so that they can experience the artistic narrow DOF images while simultaneously making available the image detail for the entire image. To enable image refocusing, one typical approach is to capture the entire light field. But this method has the drawback of a significant sacrifice in spatial resolution due to the dimensionality gap: the captured information (light field) is 4D, while the required information (focal stack) is only 3D.In this thesis, I present an imaging system that directly captures focal stacks by a sweeping focal plane. First, I describe how to synchronize focus sweeping with image capturing so that the summed DOF of a focal stack efficiently covers the entire depth range. Then, I take a customized algorithm to enable a seamless refocusing experience, even in textureless regions or with moving objects. Prototype cameras are presented to capture real space-time focal stacks. There is also an interactive refocusing viewer available online at www.focalsweep.com.

Most existing image deblurring methods are based on the estimation of blur kernels and end-to-end learning of the mapping relationship between blurred and sharp images. However, since different real-world blurred images typically have completely different blurring patterns, the performance of these methods in real image deblurring tasks is limited without explicitly modeling blurring as degradation representations. In this paper, we propose IDR2ENet, which is the Implicit Degradation Representations and Reblur Estimation Network, for real image deblurring. IDR2ENet consists of a degradation estimation process, a reblurring process, and a deblurring process. The degradation estimation process takes the real blurred image as input and outputs the implicit degradation representations estimated on it, which are used as the inputs of both reblurring and deblurring processes to better estimate the features of the blurred image. The experimental results show that whether compared with traditional or deep-learning-based deblurring algorithms, IDR2ENet achieves stable and efficient deblurring results on real blurred images.

In the blind inversion of digital images, the blur or point spread function (PSF) has to be estimated from the observed image. This general problem can be divided into several levels of difficulty depending mainly on the properties of the blur. Here, the first level of difficulty is addressed and space-invariant analytic PSF are considered. In this case, the generalized cross-validation criterion (GCV), using an AR modeling of the original image, is well known to be a robust estimator. This study use a weak constraint of smoothness on the original image and applies the GCV criterion in a myopic scheme. The problem is to choose which PSF in a set, if any, has blurred the observed image. The minimum of the GCV criterion for each candidate PSF yields a vector of estimated parameters. The actual PSF and its parameters should correspond to the lowest value of the GCV criterion. If the observed image is not blurred, the GCV criterion should attain its lowest value when the candidate PSF is reduce to a single pixel. Simulation results show that this approach yields good results on various kind of images with low signal-to-noise ratio and can discriminate between blurred and unblurred images. A near optimal value of the regularization parameter is estimated at the same time. If the effective PSF does not belong to the set of candidates, the optimization of the regularization operator, as a mean to compensate for the distance between the two PSF, is investigated.© (1998) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

This paper proposes a technique for restoring the motion blurred images. Restoration of blurred images is very important problem in tracking and identification of criminals, where image of a human face or number plate of a running vehicle taken in hit and run situation gets blurred due to relative motion between the imaging system and object/face. For restoration of motion blurred images, knowledge of the point spread function (PSF) is very important. The motion blur PSF is characterized by two parameters, namely blur direction and blur length. This paper also presents a method to identify the parameters of the PSF from the blurred and noisy images using the log spectrum of the blurred images. These parameters are used to restore the images. The experimental results demonstrate that the image can successfully be restored from the image with substantial amount of natural and artificial blur.

Atmospheric turbulence often limits the performance of long-range imaging systems in applications. Realistic turbulence simulations provide means to evaluate this effect and assess turbulence mitigation algorithms. Current methods typically use phase screens or turbulent point spread functions (PSFs) to simulate the image distortion and blur due to turbulence. While the first method takes long computation times, the latter requires empirical models or libraries of PSF shapes and their associated tip and tilt motion, which might be overly simplistic for some applications. In this work, an approach is evaluated which tries to avoid these issues. Generative neural networks models are able to generate extremely realistic imitations of real (image) data with a short calculation time. To treat anisoplanatic imaging for the considered application, the model output is an imitation PSF-grid that has to be applied to the input image to yield the turbulent image. Certain shape features of the model outcome can be controlled by traversing within subsets of the model input space or latent space. The use of a conditional variational autoencoder (cVAE) appears very promising to yield fast computation times and realistic PSFs and is therefore examined in this work. The cVAE is trained on field trial camera images of a remote LED array. These images are considered as grids of real PSFs. First the images are pre-processed and their PSFs properties are determined for each frame. Main goal of the cVAE is the generation of PSF-grids under conditional properties, e.g. moments of PSFs. Different approaches are discussed and employed for a qualitative evaluation of the realism of the PSF-grids generated by the trained models. A comparison of required simulation computing time is presented and further considerations regarding the simulation method are discussed.

Imaging of pressure-sensitive paint (PSP) for pressure measurement on moving surfaces is problematic due to the movement of the object within the finite exposure time of the imager, resulting in the blurring of the blade edges. The blurring problem is particularly challenging when high-sensitivity PSP with a long lifetime is used, where the long luminescence time constant of exponential light decay following a burst of excitation light energy results in blurred images. One method to ameliorate this effect is image deconvolution using a point spread function (PSF) based on an estimation of the luminescent time constant. Prior implementations of image deconvolution for PSP deblurring have relied upon a spatially invariant time constant in order to reduce computational time. However, the use of an assumed value of time constant leads to errors in the point spread function, particularly when strong pressure gradients (which cause strong spatial gradients in the decay time constant) are involved. This work introduces an iterative method of image deconvolution, where a spatially variant PSF is used. The point-by-point PSF values are found in an iterative manner, since the time constant depends on the local pressure value, which can only be found from the reduced PSP data. The scheme estimates a super-resolved spatially varying blur kernel with sub-pixel resolution without filtering the blurred image, and then restores the image using classical iterative regularization tools. A kernel-free forward model has been used to generate test images with known pressure surface maps and a varying amount of noise to evaluate the applicability of this scheme in different experimental conditions. A spinning disk setup with a grazing nitrogen jet for producing strong pressure gradients has also been used to evaluate the scheme on a real-world problem. Results including the convergence history and the effect of a regularization-iteration count are shown, along with a comparison with the previous PSP deblurring method.

Blind image deblurring, which restores a clear latent image using a single blurred image, is an ill-posed problem to fmd two unknowns: the latent image and the point spread function (PSF). In this paper, we propose a novel PSF estimation method utilizing total variation regularization, a shock filter, and the gradient degree of reliability for non-uniform blurred images. The experimental results demonstrate that our proposed method delivers the best performance of the currently available methods with respect to deblurring performance. Furthermore, the processing time for our proposed method is very short, and clear images are obtained for actual (real-life) pictures.

In this paper, we propose a blind method that rapidly restores blurred images using local patches. In this method, a portion of the blurred image is used for PSF (point spread function) estimation. In addition, we propose an automatic PSF size calculation algorithm that generates an autocorrelation map (auto map). Experimental results show that our proposed method generates accurate de-blurred images, and processing time is significantly lower than that of the conventional de-blurring method.

An image frame in image sequences, in general, suffers from degradation due to spatially varying motions. In this paper, we propose a new image degradation model for space-variant motion blur and a spatially adaptive image restoration algorithm to remove such motion blur. For the proposed image degradation model, we mathematically analyze boundary effect which arises on the border of two image segments with different motions. We extend the already proposed model for a moving object in an arbitrary direction. In order to represent the point spread function (PSF) for motion blur in an arbitrary direction, we develop a method which distributes energy of samples in the PSF into the neighboring integer grids. In order to remove the above mentioned motion blur, we propose an object-based adaptive regularized image restoration algorithm. Both in synthetically and naturally motion blurred images, the proposed image restoration algorithm gives acceptable performance in removing motion blur and, as a result, in restoring important features, such as numbers and characters, which cannot be recognized in the input blurred image.

Restoration of TDI camera images with motion distortion and blur

Platform attitude jitter is inevitable during a satellite flight. The jitter reduces the quality of push-broom optical images, resulting in geometric deformation and image blur. This paper proposes an approach that offers jitter effect compensation and quality improvement for push-broom optical images based on dynamic point spread function (PSF) estimation and iterative image restoration considering the effect of terrain relief and time delay integration (TDI). First, the attitude jitter is precisely detected and estimated by considering the terrain relief. The jitter information is then used to establish the dynamic PSF of each image line considering the changing jitter values and TDI effect. Finally, based on the constructed PSF, the image is improved by iterative restoration using the optimal-window Richardson-Lucy algorithm. The method is validated by both simulation and real data experiments. In the simulation experiment, we apply jitter with different amplitudes and frequencies to generate the degraded images and then restore the image using the proposed restoration method. The results show that the proposed method can effectively restore images affected by jitter. In addition, real data experiments are carried out with multispectral remote sensing images from the ZY-3 satellite. The results show that in addition to the improvement of the radiometric quality, the geometric quality is also significantly improved in both the across-track and the along-track directions. The experimental results validated that the proposed method outperformed other methods without considering the terrain and TDI effect.

This research was conducted on inseptisol soils, especially for plantations, with the aim of not only restoring the health and fertility of the soil due to the use of chemical fertilizers and pesticides as well as seeing the pattern of EC distribution at each depth from the center of the biohole based on the time of observation. Through controlled microbial activity, its spread through two types of biohole, namely horizontal and vertical biohole. This research observes in real time through soil parameter sensors connected to the micro controller to changes in soil acidity, infiltration rate, conductivity electrolyte level and porosity level through soil infiltration rate. Through simulations with 2 types of biohole, it can be seen the increase in EC in each depth to the time of observation in real time. From the observations of graphs and EC standards, it can be seen that the ability of the soil to provide nutrients in the root growth zone to support the schedule and distribution patterns of planting both during vegetative growth and generative growth periods. So that we will know the proper biohole distance and spacing in order to be able to provide vegetative and generative mass nutrition based on nutrient values monitored through sensors that change the analogue parameters in the micro possessor into digital information transmitted by Wi-Fi in real time. Inseptisol soils fertility simulation based on the number of microbial populations=108/cfu with variable 1: The effective depth for providing root nutrition to a depth of 30 cm with a soil fertility value or Electrolyte Conductivity/ EC 1000 uS/cm to 1900 uS/cm is reached on the 10 to 60th day. Variable 2: The effective depth for providing root nutrition is up to a depth of 24 cm with a soil fertility value or Electrolyte Conductivity/EC 1000 uS/cm to 2350 uS/cm achieved on the 10th to 60th day.

An accurate measurement of the point spread function (PSF) for the extended depth of field (EDoF) cell phone camera with CMOS sensor is very important for image processing and image restoration. But due to the coarse sampling of the PSF by CMOS sensor, the overall system including the imaging subsystem and sampling subsystem is a shift-variant system with respect to the sample-scene phase parameter within sub-pixel range. In this paper, we present the sub-pixel digital algorithm to estimate the overall camera PSF based on the measurement of a high resolution PSF of the imaging lens. The sub-pixel digital algorithm averages the shifted high resolution PSFs of the lens over one active pixel area with the assumption of uniform random distribution of point source location within one active pixel area. Then the averaged high resolution PSF is down sampled onto Bayer plane to obtain the shift-invariant overall system PSF. We applied this shift-invariant PSF for image restoration of blurred images captured with an extended depth of field camera. The processed images are compared with originally captured images. Improvement of image quality is seen.