Depth-aware deconvolution framework for physics-informed Monte Carlo deblurring in optical resolution photoacoustic microscopy

Optical microscopy, providing valuable insights at the cellular and organelle levels, has been widely recognized as an enabling biomedical technology. As the mainstays of in vivo three-dimensional (3-D) optical microscopy, single-/multi-photon fluorescence microscopy and optical coherence tomography (OCT) have demonstrated their extraordinary sensitivities to fluorescence and optical scattering contrasts, respectively. However, the optical absorption contrast of biological tissues, which encodes essential physiological/pathological information, has not yet been assessable. The emergence of biomedical photoacoustics has led to a new branch of optical microscopy optical-resolution photoacoustic microscopy (OR-PAM)1, where the optical irradiation is focused to the diffraction limit to achieve cellular1 or even subcellular2 level lateral resolution. As a valuable complement to existing optical microscopy technologies, OR-PAM brings in at least two novelties. First and most importantly, OR-PAM detects optical absorption contrasts with extraordinary sensitivity (i.e., 100%). Combining OR-PAM with fluorescence microscopy3 or with optical-scattering-based OCT4 (or with both) provides comprehensive optical properties of biological tissues. Second, OR-PAM encodes optical absorption into acoustic waves, in contrast to the pure optical processes in fluorescence microscopy and OCT, and provides background-free detection. The acoustic detection in OR-PAM mitigates the impacts of optical scattering on signal degradation and naturally eliminates possible interferences (i.e., crosstalks) between excitation and detection, which is a common problem in fluorescence microscopy due to the overlap between the excitation and fluorescence spectra. Unique for optical absorption imaging, OR-PAM has demonstrated broad biomedical applications since its invention, including, but not limited to, neurology5, 6, ophthalmology7, 8, vascular biology9, and dermatology10. In this video, we teach the system configuration and alignment of OR-PAM as well as the experimental procedures for in vivo functional microvascular imaging.

The penetration depth of ballistic optical imaging technologies is limited by light scattering. To study the effect of scattering on optical-resolution photoacoustic microscopy (OR-PAM), we divided the signals in OR-PAM into two classes: one is from the target volume defined by the optical resolution cell (Class I); the other is from the rest of the acoustic resolution cell (Class II). We developed a way to simulate the point spread function (PSF) of our OR-PAM system considering both optical illumination and acoustic detection, then used the PSF to calculate the contributions of each class of signal to the total signal at different focal depths. Our simulation results showed that: 1) The Class II signal decays much more slowly than the Class I signal; 2) The full width at half maximum (FWHM) of the PSF for the focal depth of 0.9 transport mean free path (TMFP) is not broadened much (~10%) compared with that for a clear medium; 3) Image contrast is degraded with increasing depth when there is a uniform absorption background.

Optical Coherence Tomography (OCT) has a great potential to elicit clinically useful information from tissues due to its high axial and transversal resolution. In practice, an OCT setup cannot reach to its theoretical resolution due to imperfections of its components, which make its images blurry. The blurriness is different alongside regions of image; thus, they cannot be modeled by a unique point spread function (PSF). In this paper, we investigate the use of solid phantoms to estimate the PSF of each sub-region of imaging system. We then utilize Lucy-Richardson, Hybr and total variation (TV) based iterative deconvolution methods for mitigating occurred spatially variant blurriness. It is shown that the TV based method will suppress the so-called speckle noise in OCT images better than the two other approaches. The performance of proposed algorithm is tested on various samples, including several skin tissues besides the test image blurred with synthetic PSF-map, demonstrating qualitatively and quantitatively the advantage of TV based deconvolution method using spatially-variant PSF for enhancing image quality.

An accurate measurement of the point spread function (PSF) for the extended depth of field (EDoF) cell phone camera with CMOS sensor is very important for image processing and image restoration. But due to the coarse sampling of the PSF by CMOS sensor, the overall system including the imaging subsystem and sampling subsystem is a shift-variant system with respect to the sample-scene phase parameter within sub-pixel range. In this paper, we present the sub-pixel digital algorithm to estimate the overall camera PSF based on the measurement of a high resolution PSF of the imaging lens. The sub-pixel digital algorithm averages the shifted high resolution PSFs of the lens over one active pixel area with the assumption of uniform random distribution of point source location within one active pixel area. Then the averaged high resolution PSF is down sampled onto Bayer plane to obtain the shift-invariant overall system PSF. We applied this shift-invariant PSF for image restoration of blurred images captured with an extended depth of field camera. The processed images are compared with originally captured images. Improvement of image quality is seen.

Multiplexed Superresolution CRISPR Imaging of Chromatin in Living Cells

Context. Fitting photometry algorithms can be very effective provided that an accurate model of the instrumental point spread function (PSF) is available. When high-precision time-resolved photometry is required, however, the use of point-source star images as empirical PSF models can be unsatisfactory, due to the limits in their spatial resolution. Theoretically-derived models, on the other hand, are limited by the unavoidable assumption of simplifying hypothesis, while the use of analytical approximations is restricted to regularly-shaped PSFs. Aims. This work investigates an innovative technique for space-based fitting photometry, based on the reconstruction of an empirical but properly-resolved PSF. The aim is the exploitation of arbitrary star images, including those produced under intentional defocus. The cases of both MOST and COROT, the first space telescopes dedicated to time-resolved stellar photometry, are considered in the evaluation of the effectiveness and performances of the proposed methodology. Methods. PSF reconstruction is based on a set of star images, periodically acquired and presenting relative subpixel displacements due to motion of the acquisition system, in this case the jitter of the satellite attitude. Higher resolution is achieved through the solution of the inverse problem. The approach can be regarded as a special application of super-resolution techniques, though a specialised procedure is proposed to better meet the PSF determination problem specificities. The application of such a model to fitting photometry is illustrated by numerical simulations for COROT and on a complete set of observations from MOST. Results. We verify that, in both scenarios, significantly better resolved PSFs can be estimated, leading to corresponding improvements in photometric results. For COROT, indeed, subpixel reconstruction enabled the successful use of fitting algorithms despite its rather complex PSF profile, which could hardly be modeled otherwise. For MOST, whose direct-imaging PSF is closer to the ordinary, comparison to other models or photometry techniques were carried out and confirmed the potential of PSF reconstruction in real observational conditions.

The Mid Infra Red Instrument (MIRI) is one of the four instruments onboard the James Webb Space Telescope (JWST), providing imaging, coronagraphy and spectroscopy over the 5-28 microns band. To verify the optical performance of the instrument, extensive tests were performed at CEA on the flight model (FM) of the Mid-InfraRed IMager (MIRIM) at cryogenic temperatures and in the infrared. This paper reports on the point spread function (PSF) measurements at 5.6 microns, the shortest operating wavelength for imaging. At 5.6 microns the PSF is not Nyquist-sampled, so we use am original technique that combines a microscanning measurement strategy with a deconvolution algorithm to obtain an over-resolved MIRIM PSF. The microscanning consists in a sub-pixel scan of a point source on the focal plane. A data inversion method is used to reconstruct PSF images that are over-resolved by a factor of 7 compared to the native resolution of MIRI. We show that the FWHM of the high-resolution PSFs were 5-10% wider than that obtained with Zemax simulations. The main cause was identified as an out-of-specification tilt of the M4 mirror. After correction, two additional test campaigns were carried out, and we show that the shape of the PSF is conform to expectations. The FWHM of the PSFs are 0.18-0.20 arcsec, in agreement with simulations. 56.1-59.2% of the total encircled energy (normalized to a 5 arcsec radius) is contained within the first dark Airy ring, over the whole field of view. At longer wavelengths (7.7-25.5 microns), this percentage is 57-68%. MIRIM is thus compliant with the optical quality requirements. This characterization of the MIRIM PSF, as well as the deconvolution method presented here, are of particular importance, not only for the verification of the optical quality and the MIRI calibration, but also for scientific applications.

Studying oxygen metabolism during brain ischemia has been limited by imaging modalities that have either low resolution but good tissue penetration, or high resolution requiring invasive preparations (open-skull windows). Here, we report a study of cerebrovascular oxygen metabolism in mouse focal ischemia using a novel minimally invasive, high-resolution technique known as optical-resolution photoacoustic microscopy (OR-PAM). Using OR-PAM, we serially imaged cortical vessels through the intact skull of Swiss Webster mice (n=3) before, during, and up to 72 hrs after a 1-hr intraluminal transient middle cerebral artery occlusion (MCAO). Vessels were imaged at 2 different wavelengths (532 & 563 nm) to quantify oxy-, deoxy-hemoglobin and oxygen saturation (sO 2 , Fig , A-C). Oxygen extraction fraction (OEF) was calculated from venous sO 2 values (assuming a constant arterial sO 2 of 95%). Seventy-two hrs after ischemia, animals were sacrificed and whole brains were stained with TTC to delineate infarct at the cortical surface (D). Arterioles were easily distinguished from venules in baseline images (A, mean arterial sO 2 , 87%; mean venous sO 2 , 76%). During MCAO, mean arteriolar sO 2 values did not change; however, venular sO 2 decreased precipitously (B, E), greater in regions that went on to infarct (45±2%) compared to surrounding regions (52±2%, p<0.001). Three and 7 hrs after reperfusion, venous sO 2 slowly recovered but not back to baseline levels. Twenty-four (C) and 72 hrs after reperfusion, venous sO 2 in regions of infarction increased above baseline (83%), approaching arterial sO 2 values of 90%, while venous sO2 in peri-infarct regions (71%) returned near baseline levels of 76% (E). OEF increased from 18% to 55% in regions of eventual infarction, during MCAO; following reperfusion, OEF progressively dropped below baseline levels (12%). In surrounding surviving tissue, OEF returned close to baseline levels (24%) following reperfusion (F). OR-PAM is capable of measuring oxygen metabolism at high resolution during focal ischemia. Consistent with previous PET studies in human stroke, we found OEF increased during acute ischemia; however, OEF remained elevated (above baseline values) after reperfusion, suggesting that injured tissue may have increased oxygen metabolism. OEF gradually decreased below baseline values in infarcted tissue, but returned to near-baseline values in viable peri-infarct regions.

Linear systems theory combined with Fourier transform techniques provides a powerful means to characterize electro-optical systems. We present the description and analysis of a particular scanning IR system whose characteristic Point Spread Function (PSF) is space-variant. This peculiarity of the PSF arises from the scanning geometry; the IR sensor spins as it drops toward the ground. A system impulse response, or Point Spread Function (PSF), is derived from effects including diffraction, motion blur, chromatic aberration and detector area. Other effects may be taken into account by cascading PSF's. Once the system PSF is determined, the output of the system is the convolution of the input irradiance with the PSF. Alternatively, the output is the familiar inverse Fourier transform of the product of transforms. However, the space-variance of the PSF makes the latter option impossible and the former difficult. The convolution progresses in a spiral manner, with the PSF width changing continuously. The method is tested using Fortran programs and a digital image processor. The digital images produced show expected results, revealing a marked increase in resolution as the sensor nears the ground.

We propose a new method of blind deconvolution (BD) suitable for the recovery of truncated blurred images. In practical cases we often encounter truncated images, but they are difficult to treat properly with conventional BD algorithms, since nontruncated blurred images are assumed. The proposed method is based on simulated annealing (SA), to which we introduce an adaptive masking process. After a truncated segment is cut out from the blurred image, the sharp edges surrounding the segment are removed by multiplying the initial window, whose shape is determined with the initial estimate of the point spread function (PSF). A more appropriate estimate for the PSF is obtained from the masked segment based on the SA algorithm, and according to the new estimate of the PSF, the shape of the window is varied. These procedures are iterated until we finally obtain the best possible estimate of the PSF. In the last step, the entire blurred image is deconvolved with the final estimate of the PSF to obtain the recovered image. The effectiveness of the proposed method is confirmed by computer simulation.

This paper presents an approach to detect sharp edges for estimating point spread function (PSF) of a lens. A category of PSF estimation methods detect sharp edges from low-resolution (LR) images and estimate PSF with the detected edges. Existing techniques usually rely on accurate detection of ending points of the profile normal to an edge. In practice, however, it is often very difficult to localize profiles accurately. Inaccurately localized profiles generate a poor PSF estimation. We employ the Random Sample Consensus (RANSAC) algorithm to rule out outlier points. In RANSAC, prior knowledge about a pattern shape is incorporated, and the edge points lying far away from the pattern shape will be removed. The proposed method is tested on images of saddle patterns. Experimental results show that the proposed method can robustly localize sharp edges from LR saddle pattern images and yield accurate PSF estimation.

In order to understand the pathophysiology of glaucoma, Lamina cribrosa (LC) perfusion needs to be the subject of thorough investigation. It is currently difficult to obtain high resolution images of the embedded microcapillary network of the LC using conventional imaging techniques. In this study, an optical resolution photoacoustic microscopy (OR-PAM) system was used for imaging lamina cribrosa of an ex vivo porcine eye. Extrinsic contrast agent was used to perfuse the eye via its ciliary arteries. The OR-PAM system have a lateral resolution of 4 μm and an axial resolution of 30 μm. The high resolution system could able resolve a perfused LC microcapillary network to show vascular structure within the LC thickness. OR-PAM could be a promising imaging modality to study the LC perfusion and hence could be used to elucidate the hemodynamic aspect of glaucoma.

A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. Over the last decade, a range of computational cameras have been proposed, which use various optics designs to encode and use computation to decode useful visual information. What is often missing, however, is the quantitative analysis of the relation between camera design and the captured visual information, and little systematic work has been done to evaluate and optimize these computational camera designs. While the optics of computational cameras may be quite complicated, many of them can be effectively characterized by their point spread functions (PSFs): the intensity distribution on an image sensor as a response to a point light source in a scene. This thesis explores the techniques to characterize, evaluate and optimize computational cameras via PSF engineering for computer vision tasks, including image recovery, 3D reconstruction, and image refocusing. I first explore PSF engineering techniques to recover image details from blurry images. Image blur is a problem commonly seen in photography due to a number of reasons, including defocus, lens aberration, diffraction, atmospheric turbulence, object motion, and etc. Image blur is often formulated as a convolution of the latent blur-free image and a PSF, and deconvolution (or deblurring) techniques have to be used to recover image details from a blurry region. Here, I propose a comprehensive framework of PSF evaluation for the purpose of image deblurring, in which the effects of image noise, deblurring algorithm, and the structure of natural images are all taken into account. In defocus blur, the shape of defocus PSF is largely determined by the aperture pattern of the camera lens. By using an evaluation criterion derived from the comprehensive framework, I optimize the aperture pattern to preserve many more image details when defocus occurs. Both through simulations and experiments, I demonstrate the significant improvement gained by using optimized aperture patterns. While defocus blur causes a loss in image detail, it also encodes depth information of scenes in images. A typical depth from defocus (DFD) technique computes depth from two or more images that are captured with circular aperture lenses of different focus settings. Circular apertures produce circular defocus PSFs. In this thesis, I show that the use of a circular aperture severely restricts the accuracy of DFD, and propose a comprehensive framework of PSF evaluation for depth recovery. With this framework, we can derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using a genetic algorithm and gradient descent search to arrive at a pair of high resolution apertures. The two coded apertures are found to complement each other in the scene frequencies they preserve. With this property it becomes possible to not only recover depth with greater fidelity but also to obtain a high quality all-focused image from the two defocused images. While depth recovery can significantly benefit from optimized aperture patterns, its overall performance is rigidly limited by the lens aperture's physical size. To transcend this limitation, I propose a novel depth recovery technique using an optical diffuser - referred to as depth from diffusion (DFDiff). I show that DFDiff is analogous to conventional DFD, in which the scatter angle of the diffuser determines the system's effective aperture. High precision depth estimation can be achieved by choosing a proper diffuser and no longer requires the large lenses that DFD requires. Even a consumer camera with a low-end small lens can be used to do high-precision depth estimation when coupled with an optical diffuser. In my detailed analysis of the image formation properties of a DFDiff system, I show a number of examples demonstrating greater precision in depth estimation when using DFDiff. While the finite depth of field (DOF) of a lens camera leads to defocus blur, it also produces artistic visual experience. Many of today's displays are interactive in nature, which opens up a possibility for new kind of visual representations. Users could, for example, interactively refocus images to different depths, so that they can experience the artistic narrow DOF images while simultaneously making available the image detail for the entire image. To enable image refocusing, one typical approach is to capture the entire light field. But this method has the drawback of a significant sacrifice in spatial resolution due to the dimensionality gap: the captured information (light field) is 4D, while the required information (focal stack) is only 3D.In this thesis, I present an imaging system that directly captures focal stacks by a sweeping focal plane. First, I describe how to synchronize focus sweeping with image capturing so that the summed DOF of a focal stack efficiently covers the entire depth range. Then, I take a customized algorithm to enable a seamless refocusing experience, even in textureless regions or with moving objects. Prototype cameras are presented to capture real space-time focal stacks. There is also an interactive refocusing viewer available online at www.focalsweep.com.

We present two new source extraction methods, based on Bayesian model selection and using the Bayesian Information Criterion (BIC). The first is a source detection filter, able to simultaneously detect point sources and estimate the image background. The second is an advanced photometry technique, which measures the flux, position (to sub-pixel accuracy), local background and point spread function. We apply the source detection filter to simulated Herschel-SPIRE data and show the filter's ability to both detect point sources and also simultaneously estimate the image background. We use the photometry method to analyse a simple simulated image containing a source of unknown flux, position and point spread function; we not only accurately measure these parameters, but also determine their uncertainties (using Markov-Chain Monte Carlo sampling). The method also characterises the nature of the source (distinguishing between a point source and extended source). We demonstrate the effect of including additional prior knowledge. Prior knowledge of the point spread function increase the precision of the flux measurement, while prior knowledge of the background has onlya small impact. In the presence of higher noise levels, we show that prior positional knowledge (such as might arise from a strong detection in another waveband) allows us to accurately measure the source flux even when the source is too faint to be detected directly. These methods are incorporated in SUSSEXtractor, the source extraction pipeline for the forthcoming Akari FIS far-infrared all-sky survey. They are also implemented in a stand-alone, beta-version public tool that can be obtained at http://astronomy.sussex.ac.uk/$\sim$rss23/sourceMiner\_v0.1.2.0.tar.gz

The three-dimensional (3D) imaging performance of a cone-beam computed tomography (CBCT) system can be quantitatively characterized by its 3D point spread function (PSF). To avoid the pitfalls associated with 3D PSF measurement through the use of a micro point phantom, we adopt an edge-based technique, which iteratively blurs a step edge into a spread edge with a Gaussian blurring kernel. In experiment, a small Teflon solid ball (diameter ~6mm) is used to provide step edges along the scanlines across the ball center, in terms of x-ray linear attenuation coefficient. Correspondingly, the spread edge profiles are extracted from the digital volume ball reconstructed by a CBCT system. From a spread edge profile, a step edge can be established by a rectification procedure (essentially an edge identification). A 1D PSF along a scanline is modeled as a Gaussian distribution, which is determined by iterative edge-blurring technique. The 1D PSFs on a cross section of the ball constitute a 2D PSF, and the 2D PSFs at three orthogonal cross sections are used to represent the 3D PSF at the position of the ball. By repositioning the ball phantom and repeating the procedure, we measured the 3D PSFs at 126 positions over half of the support space. Experiment shows that a CBCT system is of spatial variance and anisotropy in terms of FWHMs (full width at half maximum) of the blurring kernel. Both numerical and graphical presentations of PSF results are provided. As a result, the FWHM of the 3D PSF in our CBCT system varies in the range between 0.66 mm and 1.39 mm.