Learning Time-multiplexed phase-coded apertures for snapshot spectral-depth imaging.

A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. Over the last decade, a range of computational cameras have been proposed, which use various optics designs to encode and use computation to decode useful visual information. What is often missing, however, is the quantitative analysis of the relation between camera design and the captured visual information, and little systematic work has been done to evaluate and optimize these computational camera designs. While the optics of computational cameras may be quite complicated, many of them can be effectively characterized by their point spread functions (PSFs): the intensity distribution on an image sensor as a response to a point light source in a scene. This thesis explores the techniques to characterize, evaluate and optimize computational cameras via PSF engineering for computer vision tasks, including image recovery, 3D reconstruction, and image refocusing. I first explore PSF engineering techniques to recover image details from blurry images. Image blur is a problem commonly seen in photography due to a number of reasons, including defocus, lens aberration, diffraction, atmospheric turbulence, object motion, and etc. Image blur is often formulated as a convolution of the latent blur-free image and a PSF, and deconvolution (or deblurring) techniques have to be used to recover image details from a blurry region. Here, I propose a comprehensive framework of PSF evaluation for the purpose of image deblurring, in which the effects of image noise, deblurring algorithm, and the structure of natural images are all taken into account. In defocus blur, the shape of defocus PSF is largely determined by the aperture pattern of the camera lens. By using an evaluation criterion derived from the comprehensive framework, I optimize the aperture pattern to preserve many more image details when defocus occurs. Both through simulations and experiments, I demonstrate the significant improvement gained by using optimized aperture patterns. While defocus blur causes a loss in image detail, it also encodes depth information of scenes in images. A typical depth from defocus (DFD) technique computes depth from two or more images that are captured with circular aperture lenses of different focus settings. Circular apertures produce circular defocus PSFs. In this thesis, I show that the use of a circular aperture severely restricts the accuracy of DFD, and propose a comprehensive framework of PSF evaluation for depth recovery. With this framework, we can derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using a genetic algorithm and gradient descent search to arrive at a pair of high resolution apertures. The two coded apertures are found to complement each other in the scene frequencies they preserve. With this property it becomes possible to not only recover depth with greater fidelity but also to obtain a high quality all-focused image from the two defocused images. While depth recovery can significantly benefit from optimized aperture patterns, its overall performance is rigidly limited by the lens aperture's physical size. To transcend this limitation, I propose a novel depth recovery technique using an optical diffuser - referred to as depth from diffusion (DFDiff). I show that DFDiff is analogous to conventional DFD, in which the scatter angle of the diffuser determines the system's effective aperture. High precision depth estimation can be achieved by choosing a proper diffuser and no longer requires the large lenses that DFD requires. Even a consumer camera with a low-end small lens can be used to do high-precision depth estimation when coupled with an optical diffuser. In my detailed analysis of the image formation properties of a DFDiff system, I show a number of examples demonstrating greater precision in depth estimation when using DFDiff. While the finite depth of field (DOF) of a lens camera leads to defocus blur, it also produces artistic visual experience. Many of today's displays are interactive in nature, which opens up a possibility for new kind of visual representations. Users could, for example, interactively refocus images to different depths, so that they can experience the artistic narrow DOF images while simultaneously making available the image detail for the entire image. To enable image refocusing, one typical approach is to capture the entire light field. But this method has the drawback of a significant sacrifice in spatial resolution due to the dimensionality gap: the captured information (light field) is 4D, while the required information (focal stack) is only 3D.In this thesis, I present an imaging system that directly captures focal stacks by a sweeping focal plane. First, I describe how to synchronize focus sweeping with image capturing so that the summed DOF of a focal stack efficiently covers the entire depth range. Then, I take a customized algorithm to enable a seamless refocusing experience, even in textureless regions or with moving objects. Prototype cameras are presented to capture real space-time focal stacks. There is also an interactive refocusing viewer available online at www.focalsweep.com.

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

A new code aperture design framework for multiframe code aperture snapshot spectral imaging (CASSI) system is presented. It aims at the optimization of code aperture sets such that a group of compressive spectral measurements is constructed, each with information from a specific subset of bands. A matrix representation of CASSI is introduced that permits the optimization of spectrally selective code aperture sets. Furthermore, each code aperture set forms a matrix such that rank minimization is used to reduce the number of CASSI shots needed. Conditions for the code apertures are identified such that a restricted isometry property in the CASSI compressive measurements is satisfied with higher probability. Simulations show higher quality of spectral image reconstruction than that attained by systems using Hadamard or random code aperture sets.

Contribution of time of flight and point spread function modeling to the performance characteristics of the PET/CT Biograph mCT scanner

Atmospheric turbulence often limits the performance of long-range imaging systems in applications. Realistic turbulence simulations provide means to evaluate this effect and assess turbulence mitigation algorithms. Current methods typically use phase screens or turbulent point spread functions (PSFs) to simulate the image distortion and blur due to turbulence. While the first method takes long computation times, the latter requires empirical models or libraries of PSF shapes and their associated tip and tilt motion, which might be overly simplistic for some applications. In this work, an approach is evaluated which tries to avoid these issues. Generative neural networks models are able to generate extremely realistic imitations of real (image) data with a short calculation time. To treat anisoplanatic imaging for the considered application, the model output is an imitation PSF-grid that has to be applied to the input image to yield the turbulent image. Certain shape features of the model outcome can be controlled by traversing within subsets of the model input space or latent space. The use of a conditional variational autoencoder (cVAE) appears very promising to yield fast computation times and realistic PSFs and is therefore examined in this work. The cVAE is trained on field trial camera images of a remote LED array. These images are considered as grids of real PSFs. First the images are pre-processed and their PSFs properties are determined for each frame. Main goal of the cVAE is the generation of PSF-grids under conditional properties, e.g. moments of PSFs. Different approaches are discussed and employed for a qualitative evaluation of the realism of the PSF-grids generated by the trained models. A comparison of required simulation computing time is presented and further considerations regarding the simulation method are discussed.

A fast and flexible algorithm for microstructure reconstruction combining simulated annealing and deep learning

Including accurate modeling of the point spread function (PSF) in positron emission tomography (PET) reconstruction algorithms results in improvements in image spatial resolution and contrast. In this work, we sampled the PSF in our first-generation radio-frequency brain dedicated PET insert for simultaneous PET/MR imaging using a 100 µCi NEMA standard 250 µm diameter Na-22 point source at 13 different positions within a subsection of the system field of view (FoV). The acquired list mode data was converted into the canonical sinogram format from which the spatial positioning of the source and standard deviations were calculated. The subset was then used to extrapolate the PSF for the full system FoV. This model was then fed as an input parameter into a graphical processing unit based ordered subset expectation maximization (OSEM) reconstruction algorithm and used to generate reconstructed images with and without spatially varying PSF modeling for the Na-22 point source and a Hoffman brain phantom. Results indicate that for point source reconstruction, the FWHM of the horizontal profile of the point source is smaller with spatially variant PSF especially closer to the edges. Effect of spatially varying PSF modeling is also presented with Hoffman phantom reconstruction and CNR value has increased with spatially varying PSF.

Coded aperture has been applied to short wavelength imaging (e.g., gamma-ray), and it suffers from diffraction and interference for taking longer wavelength images. This paper investigates an interleaving and sparse random (ISR) coded aperture to reduce the impact of diffraction and interference for visible imaging. The interleaving technique treats coded aperture as a combination of many small replicas to reduce the diffraction effects and to increase the angular resolution. The sparse random coded aperture reduces the interference effects by increasing the separations between adjacent open elements. These techniques facilitate the analysis of the imaging model based only on geometric optics. Compressed sensing is applied to recover the coded image by coded aperture, and a physical prototype is developed to examine the proposed techniques.

Clinical evaluation of whole-body oncologic PET with time-of-flight and point-spread function for the hybrid PET/MR system

BackgroundImage reconstruction algorithm is one of the important factors affecting the quantitative parameters of PET/CT. The purpose of this study was to investigate the effects of time of flight (TOF) and point spread function (PSF) on quantitative parameters of lung lesions in 18F-FDG PET/CT.MethodsThis retrospective study evaluated 60 lung lesions in 39 patients who had undergone 18F-fluoro-deoxy-glucose (FDG) PET/CT. All lesions larger than 10 mm in diameter were included in the study. The PET data were reconstructed with a baseline ordered-subsets expectation–maximization (OSEM) algorithm, OSEM + PSF, OSEM + TOF and OSEM + TOF + PSF respectively. The differences of maximum standard uptake value (SUVmax), mean standard uptake value (SUVmean), metabolic tumor volume (MTV), total lesion glycolysis (TLG)and signal to noise ratio (SNR)were compared among different reconstruction algorithms.ResultsCompared with OSEM reconstruction, using OSEM + TOF + PSF increased SUVmean and SUVmax by 23.73% and 22.71% respectively, and SNR increased by 70.18%, MTV decreased by 23.84% (p < 0.01). The percentage difference was significantly higher in smaller lesions (diameter 10–22 mm) than in larger lesions (diameter 23–44 mm), and significantly higher in low contrast lesions (SNR ≤ 15.31) than in high contrast lesions (SNR > 15.31). The difference of TLG among various reconstruction algorithms is relatively small, the highest value is − 6.48% of OSEM + TOF + PSF, and the lowest value is 0.81% of OSEM + TOF.ConclusionTOF and PSF significantly affected the quantitative parameters of lung lesions in 18F-FDG PET/CT. OSEM + TOF + PSF can significantly increased SUVmax, SUVmean and SNR, and significantly reduce MTV, especially in small lesions and low contrast lesions. TLG can be relatively stable in different reconstruction algorithms.

Direct exoplanet detections are limited by the speckle noise of the point spread function (PSF). This noise can be reduced by subtracting PSF images obtained simultaneously in adjacent narrow spectral bands using a multichannel camera (MCC). Experiments have shown that speckle attenuation performances are severely degraded by differential optical aberrations between channels that decorrelate the PSFs of the different spectral bands. We present a new technique which can greatly alleviate this problem: the introduction of a holographic diffuser at the focal plane of the MCC. The holographic diffuser converts the PSF image into an incoherent illumination scene that is then re-imaged with the MCC. This imaging process is equivalent to a convolution of the scene with the PSF of each channel of the MCC. The optical aberrations in the MCC then affect only the convolution kernel of each channel and not the PSF globally, resulting in more correlated images. We report laboratory measurements with a dual channel prototype (1.575 &#956;m and 1.625 &#956;m) to validate this approach. We achieved a speckle noise suppression factor of 12-14, which is ~4-6 times better than what has been achieved by existing MCCs.

Coded aperture is a promising approach for capturing the 4-D light field (LF), in which the 4-D data are compressively modulated into 2-D coded measurements that are further decoded by reconstruction algorithms. The bottleneck lies in the reconstruction algorithms, resulting in rather limited reconstruction quality. To tackle this challenge, we propose a novel learning-based framework for the reconstruction of high-quality LFs from acquisitions via learned coded apertures. The proposed method incorporates the measurement observation into the deep learning framework elegantly to avoid relying entirely on data-driven priors for LF reconstruction. Specifically, we first formulate the compressive LF reconstruction as an inverse problem with an implicit regularization term. Then, we construct the regularization term with an efficient deep spatial-angular convolutional sub-network to comprehensively explore the signal distribution free from the limited representation ability and inefficiency of deterministic mathematical modeling. Experimental results show that the reconstructed LFs not only achieve much higher PSNR/SSIM but also preserve the LF parallax structure better, compared with state-of-the-art methods on both real and synthetic LF benchmarks. In addition, experiments show that our method is efficient and robust to noise, which is an essential advantage for a real camera system. The code is publicly available at https://github.com/angmt2008/LFCA .

The main goals of research in medical and biomedical imaging are to create €œbetter€ imaging systems, more accurate reconstructions, and develop methods of image processing that utilize the most important information present in an image or set of images, for accurate, timely, and cost-effective diagnosis and treatment of disease. How well desired information can be extracted best serves to define image quality and consequently the performance of imaging systems. Furthermore, as we deepen our understanding of medical images, new systems and new methods of image processing and image display may help optimize performance and assist radiologists in their challenge for accurate diagnostics. Image quality is thus best measured by the performance of an observer on specific tasks [1]. The observer may be human such as a physician making a diagnosis, or a mathematical model such as an ideal observer [2], or a computer algorithm. Specific tasks may be the detection of a lesion in a chest x ray [3], the estimation of the percent stenosis of a detected aneurysm [4], or the registration of images such as the superimposition of anatomical atlases on patient data [5]. Regardless of the task considered, however, methods to assess image quality based on task performance are most important. Traditionally, the conception of improved imaging systems in medical imaging has been accomplished by designing and constructing proposed systems and characterizing them from an engineering point of view, by reporting parameters such as resolution, modulation transfer function, and pixel signal-to-noise ratio. Naturally, such evaluation is not sufficient to predict the performance that the system will have in the clinic, given that it does not take into account either information about the types of objects being imaged or the tasks being carried out. To fully assess the system, clinical trials based on specific tasks must be conducted. While clinical trials are required for final system assessment, the reliance on clinical trials for system design and optimization has severe limitations. First, the construction of proposed systems is costly and time consuming. In addition, if the system requires further optimization, it may take multiple trials and errors to find the best set of parameters and tradeoff to satisfy required task-based image-quality criteria. Finally, it is nearly impossible to know the anatomical underlying structures of the acquired clinical images unless the diagnosis is verified by physical examination. Instead, disease states are most often estimated by interpreting the images and looking for correlation with previous cases established statistically as well. The approach of building systems and conducting trial-and-error changes based on clinical images thus constitutes a highly impractical means of optimizing imaging systems.

Age-at-Death Estimation for Modern Populations in Mexico and Puerto Rico through the Use of 3D Laser Scans of the Pubic Symphysis.

This thesis presents a number of studies into Single Photon Emission Imaging (SPEI) that can be grouped into three main sections: Monte Carlo Radiation Transport Modelling, Development of a Hybrid SPEI System and Fundamental Study of Advanced Compton Imaging. The first section of this thesis, Monte Carlo Radiation Transport Modelling, presents an overview of the Monte Carlo radiation transport modelling toolkit Geant4 and a newly derived low energy Compton scattering model, the Monash University Compton scattering model. This Compton scattering model was developed to address the limitation present in the majority of Monte Carlo bound atomic electron Compton scattering models: incorrect determination of the ejected direction of Compton electrons due to the non-zero momentum of the bound atomic electron. A theoretical foundation that ensures the conservation of energy and momentum in the relativistic impulse approximation was utilised to develop energy and directional algorithms for both the scattered photon and ejected Compton electron from first principles. Assessment of this model was undertaken in two steps: comparison with respect to two Compton scattering classes of Geant4 adapted from Ribberfors' work, and experimental comparison with respect to Compton electron kinetic energy spectra obtained from the Compton scattering of 662 keV photons off the K-shell of gold. It was shown that this new Compton scattering model was a viable replacement for the majority of computational models that have been adapted from Ribberfors' work. Additionally, this model was shown to be able to reproduce the Compton scattering triply differential cross-section Compton electron kinetic energy spectra of 662 keV photons K-shell scattering off of gold to within experimental uncertainty. The second section of this thesis, Development of a Hybrid SPEI System, presents the development of a novel hybrid collimated SPEI system: the Pixelated Emission Detector for RadiOisotopes (PEDRO). The PEDRO is a conceptual proof of principle hybrid SPEI system that was designed to explore and quantify the relationship between spatial resolution and sensitivity, inherent in SPEI, over the energy range of 30 keV to 511 keV. This system was originally intended to be constructed from a Compton camera stack located behind a coded mask composed of a mix of pinholes, slats and/or open areas. A total of three studies are presented in this section that outline the development of: 1) a Geant4 application to be used for optimisation of PEDRO with respect to a robust metric, 2) an automated routine for the optimisation of large-area slits in the outer regions of a coded mask for PEDRO which has a central region allocated for pinholes, and 3) a novel experimentally motivated image deblurring technique for multi-plane Gamma cameras such as PEDRO. These three studies illustrated two main points: 1) it may indeed be possible to overcome the trade-off between spatial resolution and sensitivity inherent in SPEI through hybrid Compton-mechanical collimation, and 2) the multi-plane nature of hybrid collimation systems can be capitilised on to reduce the level of image blurring associated with collimator opening geometry and, in turn, improve recovered image quality. Finally, the third section of this thesis, Fundamental Study of Advanced Compton Imaging, presents a preliminary study into the fundamental limits of recoil electron tracking enhanced Compton collimation, or Advanced Compton Imaging, for photon energies below 2 MeV. A custom Geant4 application of an idealised detection system was developed and four different detection materials were tested over a wide range of electron tracking resolutions. Increased image performance and point source convergence was observed with respect to standard Compton imaging regardless of recoil electron tracking accuracy in all tested materials across the investigated energy range. Additionally, the rate of point source convergence with respect to standard Compton imaging was discovered to be maximised for electron tracking accuracies of 45° to 60° Full Width at Tenth Maximum (FWTM) for photons of incident energy greater than 500 keV. Further study with more detailed simulation and image recovery frameworks is required to assess the validity of these observed trends.