Simultaneous spectral differential imaging with a focal plane holographic diffuser

Generally, blind super-resolution with unknown blurs is treated as an optimization problem. This involves a cost function, composed of terms accounting for changes in image and point spread function (PSF), which usually undergoes regularization due to the ill-posedness of the problem. In this paper, we introduce a novel regularization term for the PSF such that the spectral change in the image caused by degradation is also included. This is based on the fact that the presence of PSF in images affects the frequency component concentration. This cost function is optimized with respect to the image and the PSF in an alternating manner. Experiment results show that the proposed method is effective based on an objective evaluation method and that its PSF estimation accuracy is competitive in comparison with the recently proposed parametric method.

A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. Over the last decade, a range of computational cameras have been proposed, which use various optics designs to encode and use computation to decode useful visual information. What is often missing, however, is the quantitative analysis of the relation between camera design and the captured visual information, and little systematic work has been done to evaluate and optimize these computational camera designs. While the optics of computational cameras may be quite complicated, many of them can be effectively characterized by their point spread functions (PSFs): the intensity distribution on an image sensor as a response to a point light source in a scene. This thesis explores the techniques to characterize, evaluate and optimize computational cameras via PSF engineering for computer vision tasks, including image recovery, 3D reconstruction, and image refocusing. I first explore PSF engineering techniques to recover image details from blurry images. Image blur is a problem commonly seen in photography due to a number of reasons, including defocus, lens aberration, diffraction, atmospheric turbulence, object motion, and etc. Image blur is often formulated as a convolution of the latent blur-free image and a PSF, and deconvolution (or deblurring) techniques have to be used to recover image details from a blurry region. Here, I propose a comprehensive framework of PSF evaluation for the purpose of image deblurring, in which the effects of image noise, deblurring algorithm, and the structure of natural images are all taken into account. In defocus blur, the shape of defocus PSF is largely determined by the aperture pattern of the camera lens. By using an evaluation criterion derived from the comprehensive framework, I optimize the aperture pattern to preserve many more image details when defocus occurs. Both through simulations and experiments, I demonstrate the significant improvement gained by using optimized aperture patterns. While defocus blur causes a loss in image detail, it also encodes depth information of scenes in images. A typical depth from defocus (DFD) technique computes depth from two or more images that are captured with circular aperture lenses of different focus settings. Circular apertures produce circular defocus PSFs. In this thesis, I show that the use of a circular aperture severely restricts the accuracy of DFD, and propose a comprehensive framework of PSF evaluation for depth recovery. With this framework, we can derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using a genetic algorithm and gradient descent search to arrive at a pair of high resolution apertures. The two coded apertures are found to complement each other in the scene frequencies they preserve. With this property it becomes possible to not only recover depth with greater fidelity but also to obtain a high quality all-focused image from the two defocused images. While depth recovery can significantly benefit from optimized aperture patterns, its overall performance is rigidly limited by the lens aperture's physical size. To transcend this limitation, I propose a novel depth recovery technique using an optical diffuser - referred to as depth from diffusion (DFDiff). I show that DFDiff is analogous to conventional DFD, in which the scatter angle of the diffuser determines the system's effective aperture. High precision depth estimation can be achieved by choosing a proper diffuser and no longer requires the large lenses that DFD requires. Even a consumer camera with a low-end small lens can be used to do high-precision depth estimation when coupled with an optical diffuser. In my detailed analysis of the image formation properties of a DFDiff system, I show a number of examples demonstrating greater precision in depth estimation when using DFDiff. While the finite depth of field (DOF) of a lens camera leads to defocus blur, it also produces artistic visual experience. Many of today's displays are interactive in nature, which opens up a possibility for new kind of visual representations. Users could, for example, interactively refocus images to different depths, so that they can experience the artistic narrow DOF images while simultaneously making available the image detail for the entire image. To enable image refocusing, one typical approach is to capture the entire light field. But this method has the drawback of a significant sacrifice in spatial resolution due to the dimensionality gap: the captured information (light field) is 4D, while the required information (focal stack) is only 3D.In this thesis, I present an imaging system that directly captures focal stacks by a sweeping focal plane. First, I describe how to synchronize focus sweeping with image capturing so that the summed DOF of a focal stack efficiently covers the entire depth range. Then, I take a customized algorithm to enable a seamless refocusing experience, even in textureless regions or with moving objects. Prototype cameras are presented to capture real space-time focal stacks. There is also an interactive refocusing viewer available online at www.focalsweep.com.

It is important to measure the spatial performance of an in-flight satellite sensor to determine if any spatial resolution degradations occur over time; then some compensation methods can be adopted to improve the image quality. In satellite on-orbit measurements of spatial resolution, the performance of the sensors can be objectively evaluated through the point spread function (PSF) or modulation transfer function (MTF). The well-known image-based PSF derivation method of sampling along a slanted linear step structure from the degraded image is widely used in the satellite image restoration area. Due to the non-uniformity of the linear ground objects, generally, the operators have to pick out good profiles from all the profiles along the edge in order to increase the estimation accuracy. However, only some qualitative criteria have been proposed in the literature to instruct researchers how to pick out good profiles. In this article, we propose a method to quantitatively measure the homogeneity and contrast of the candidate linear structures in order to extract the optimal profiles. This method first automatically detects and locates the slanted linear step edge object in the image. Then it measures the homogeneity and the contrast at each location along the edge by using two adjacent square regions on both sides of the linear step structure. The locations that have high measurement values are regarded as the optimal locations. Finally, the optimal profiles can be extracted at these edge locations to derive an accurate PSF estimation by differentiation. This method makes an automatic optimal step edge detection method for PSF derivation to be practical. Experiment results on simulated and real images are provided. This article shows that using only the ‘optimal’ profiles can play a real and important role and can achieve a good PSF estimation result.

The Mid Infra Red Instrument (MIRI) is one of the four instruments onboard the James Webb Space Telescope (JWST), providing imaging, coronagraphy and spectroscopy over the 5-28 microns band. To verify the optical performance of the instrument, extensive tests were performed at CEA on the flight model (FM) of the Mid-InfraRed IMager (MIRIM) at cryogenic temperatures and in the infrared. This paper reports on the point spread function (PSF) measurements at 5.6 microns, the shortest operating wavelength for imaging. At 5.6 microns the PSF is not Nyquist-sampled, so we use am original technique that combines a microscanning measurement strategy with a deconvolution algorithm to obtain an over-resolved MIRIM PSF. The microscanning consists in a sub-pixel scan of a point source on the focal plane. A data inversion method is used to reconstruct PSF images that are over-resolved by a factor of 7 compared to the native resolution of MIRI. We show that the FWHM of the high-resolution PSFs were 5-10% wider than that obtained with Zemax simulations. The main cause was identified as an out-of-specification tilt of the M4 mirror. After correction, two additional test campaigns were carried out, and we show that the shape of the PSF is conform to expectations. The FWHM of the PSFs are 0.18-0.20 arcsec, in agreement with simulations. 56.1-59.2% of the total encircled energy (normalized to a 5 arcsec radius) is contained within the first dark Airy ring, over the whole field of view. At longer wavelengths (7.7-25.5 microns), this percentage is 57-68%. MIRIM is thus compliant with the optical quality requirements. This characterization of the MIRIM PSF, as well as the deconvolution method presented here, are of particular importance, not only for the verification of the optical quality and the MIRI calibration, but also for scientific applications.

This study addresses the blind image deconvolution which uses only blurred image and less point spread function (PSF) information to restore the original image. To identify the blind image it is a very important step for restoring the image. Therefore, the first step is to look for PSF model. In this paper, particle swarm optimization (PSO) is utilized to seek the unknown PSF. The objective function is based on the wavelet transform. It can identify the parameters of PSF exactly. Finally, the feasibility and validity of proposed algorithm are demonstrated by several simulations.

At present time methods of optical encryption are actively developed. The majority of existing methods of optical encryption use not only light intensity distribution, easily registered with photosensors, but also its phase distribution which require application of complex holographic schemes in conjunction with spatially coherent monochromatic illumination. This leads to complex optical schemes and low decryption quality. To eliminate these disadvantages it is possible to implement optical encryption using spatially incoherent monochromatic illumination which requires registration of light intensity distribution only. Encryption is accomplished by means of optical convolution of image of scene to be encrypted and encryption diffractive optical element (DOE) point spread function (PSF) which serves as encryption key. Encryption process is described as follows. Scene is illuminated with spatially-incoherent monochromatic light. In the absence of encryption DOE lens forms image of scene in photosensor plane. DOE serves as encryption element, its PSF - encryption key. Light passing through DOE forms convolution of object image and DOE PSF. Registered by photosensor convolution is encrypted image. Decryption was conducted numerically on computer by means of inverse filtration with regularization. Kinoforms were used as encryption DOE because they have single diffraction order. Two liquid crystal (LC) spatial light modulators (SLM) were used to implement dynamic digital information input and dynamic encryption key change. As input scene amplitude LC SLM HoloEye LC2002 with 800×600 pixels 32×32 μm² and 256 gray levels was used. To image synthesized encryption kinoforms phase LC SLM HoloEye PLUTO VIS with 1920×1080 pixels 8×8 μm² and 256 phase levels was used. Set of test images was successfully optically encrypted and then numerically decrypted. Encrypted images contents are hidden. Decrypted images despite quite high noise levels are positively recognizable. Results of optical encryption and numerical decryption are presented.

Recognition of the types of aberrations corresponding to individual Zernike functions were carried out from the pattern of the intensity of the point spread function (PSF) outside the focal plane using convolutional neural networks. The PSF intensity patterns outside the focal plane are more informative in comparison with the focal plane even for small values/magnitudes of aberrations. The mean prediction errors of the neural network for each type of aberration were obtained for a set of 8 Zernike functions from a dataset of 2 thousand pictures of out-of-focal PSFs. As a result of training, for the considered types of aberrations, the obtained averaged absolute errors do not exceed 0.0053, which corresponds to an almost threefold decrease in the error in comparison with the same result for focal PSFs.

Three-dimensional images of confocal laser scanning microscopy suffer from a depth-variant blur, due to refractive index mismatch between the different mediums composing the system as well as the specimen, leading to optical aberrations. Our goal is to develop an image restoration method for 3D confocal microscopy taking into account the blur variation with depth. The difficulty is that optical aberrations depend on the refractive index of the biological specimen. The depth-variant blur function or the Point Spread Function (PSF) is thus different for each observation. A blind or semi-blind restoration method needs to be developed for this system. For that purpose, we use a previously developed algorithm for the joint estimation of the specimen function (original image) and the 3D PSF, the continuously depth-variant PSF is approximated by a convex combination of a set of space-invariant PSFs taken at different depths. We propose to add to that algorithm a pupil-phase constraint for the PSF estimation, given by the the optical instrument geometry. We thus define a blind estimation algorithm by minimizing a regularized criterion in which we integrate the Gerchberg-Saxton algorithm allowing to include these physical constraints. We show the efficiency of this method relying on some numerical tests.

An accurate measurement of the point spread function (PSF) for the extended depth of field (EDoF) cell phone camera with CMOS sensor is very important for image processing and image restoration. But due to the coarse sampling of the PSF by CMOS sensor, the overall system including the imaging subsystem and sampling subsystem is a shift-variant system with respect to the sample-scene phase parameter within sub-pixel range. In this paper, we present the sub-pixel digital algorithm to estimate the overall camera PSF based on the measurement of a high resolution PSF of the imaging lens. The sub-pixel digital algorithm averages the shifted high resolution PSFs of the lens over one active pixel area with the assumption of uniform random distribution of point source location within one active pixel area. Then the averaged high resolution PSF is down sampled onto Bayer plane to obtain the shift-invariant overall system PSF. We applied this shift-invariant PSF for image restoration of blurred images captured with an extended depth of field camera. The processed images are compared with originally captured images. Improvement of image quality is seen.

We study the possibility of using quadrupole moments of auto-convolved galaxy images to measure cosmic shear. The autoconvolution of an image corresponds to the inverse Fourier transformation of its power spectrum. The new method has the following advantages: the smearing effect due to the point-spread function (PSF) can be corrected by subtracting the quadrupole moments of the auto-convolved PSF; the centroid of the auto-convolved image is trivially identified; the systematic error due to noise can be directly removed in Fourier space; the PSF image can also contain noise, the effect of which can be similarly removed. With a large ensemble of simulated galaxy images, we show that the new method can reach a sub-percent level accuracy under general conditions, albeit with increasingly large stamp size for galaxies of less compact profiles.

During spatial remote sensing imaging procedure, combined degradation factors conduce to Gaussian image blurring. The precondition of restoring the degraded image is to estimate point spread function (PSF) of the imaging system as precisely as possible. Because the depredating processes are quite complex, the transfer function of the degraded system is often completely or partly unknown, which makes it quite difficult to identify the precise PSF. Considering the similarity between the quantum process and imaging process in the probability and statistics fields, a novel algorithm is proposed by using multilayer feed-forward back-propagation quantum neural network (QBPNN) to estimate PSF of the Gaussian degraded imaging system. Different from the classical artificial neural network (ANN), 2 adjustable parameters of weight connection coefficient and phase coefficient are introduced in its quantum neurons used in learning stage. By establishing different training sets, this estimation method can overcome the limitation in the dependence on initial values and large amount of computation. Test results show that this method can achieve higher precision, faster convergence and stronger generalization ability comparing with the traditional PSF estimation results.

Accurate 3D Fluorophore Localization using Phase Retrieved Pupil Functions

Inverse Synthetic Aperture Radar (ISAR) imaging is one of the most sophisticated methods to obtain information about the scattering or radiation properties of a flnite sized object. The idea is to process the scattered or radiated flelds coherently over a certain frequency bandwidth and over a certain angular range in order to generate the image. In a simulation based approach, this procedure can be considerably simplifled, if the source currents are known (either real or equivalent) and if a bistatic image is desired. By inserting the radiation integral into the imaging integral and by interchanging the integration orders, the imaging point spread function can be generated and the image formation is reduced to a convolution of the point spread function with the current distribution. A concise formulation of this well-known methodology is presented together with a discussion of important properties. Various examples of 2D and 3D images for complex metallic objects such as automobiles are shown, which have been obtained from the surface currents of a Shooting and Bouncing Rays (SBR) fleld solver.

The main inherent property of medical ultrasound imaging is speckle noise which generally obscures and reduces the diagnostic image resolution and contrast. Consequently, the substantial improvement of ultrasound images is an important prerequisite, whenever ultrasound is used as one of the most utilized diagnostic modalities. The problem of reconstruction of ultrasound images by means of blind deconvolution has long been recognized as a prerequisite in ultrasound imaging processing in the recent decades. Recently, the Total Variation (TV) regularization method has become extremely effective approach for image reconstruction, especially for restoring edges of the blurring image. In this paper, we present a new blind iterative TV deconvolution algorithm for reconstructing ultrasound images from blurry and noisy observations. First, it proposes the initial estimation of the point-spread function (PSF) based on a generalization of two-dimensional homomorphic filtering in cepstrum domain. It is demonstrated that the initial PSF can be effectively estimated by applying a proper smoothing low-pass filtering in cepstrum domain. Second, it introduces a novel blind iterative TV deconvolution which is derived from an alternating minimization algorithm. Fast Fourier Transform (FFT) is used in the pre-iteration computation. The innovative blind deconvolution is based on either concurrent or successive estimation of the PSF function and the image of interest. The iterative scheme is devised to recover the image and simultaneously identify the PSF function. The estimated PSF and restored image will be close to real values in the subsequent iterative deconvolution. Final, tests on phantom and clinical images have proven our novel blind iterative TV deconvolution gives more positive results of higher spatial resolution and better defined tissue structures than other deconvolution methods.

Computational imaging technology can capture extra information at the sensor and can be used for various photographic applications, including imaging with extended depth of field or depth extraction for 3D applications. The depth estimation from a single captured photograph can be achieved through a phase coded lens and image processing. In this paper, we propose a new method to design a phase coded lens, using a blur metric (BM) as the design criterion. Matlab and Zemax are used for the co-optimization of optical coding and digital image process. The purpose of the design is to find a curve for which the BM changes continuously and seriously within a distance range. We verified our approach by simulation, and got a axial symmetric phase mask as the coded lens. By using a pseudo-random pattern which contains uniform black and white patches as the input image, and the on-axis point spread function (PSF) calculated from Zemax, we can evaluate the BM of the simulated image which is convoluted by the pseudo-random pattern and PSF. In order to ensure the BM curve evaluated from the on-axis PSF represents the result of the whole field of view, the PSF is also optimized to get high off-axis similarity.