Study of Image Classification Accuracy with Fourier Ptychography

High-throughput quantitative phase imaging (QPI) is essential to cellular phenotypes characterization as it allows high-content cell analysis and avoids adverse effects of staining reagents on cellular viability and cell signaling. Among different approaches, Fourier ptychographic microscopy (FPM) is probably the most promising technique to realize high-throughput QPI by synthesizing a wide-field, high-resolution complex image from multiple angle-variably illuminated, low-resolution images. However, the large dataset requirement in conventional FPM significantly limits its imaging speed, resulting in low temporal throughput. Moreover, the underlying theoretical mechanism as well as optimum illumination scheme for high-accuracy phase imaging in FPM remains unclear. Herein, we report a high-speed FPM technique based on programmable annular illuminations (AIFPM). The optical-transfer-function (OTF) analysis of FPM reveals that the low-frequency phase information can only be correctly recovered if the LEDs are precisely located at the edge of the objective numerical aperture (NA) in the frequency space. By using only 4 low-resolution images corresponding to 4 tilted illuminations matching a 10×, 0.4 NA objective, we present the high-speed imaging results of in vitro Hela cells mitosis and apoptosis at a frame rate of 25 Hz with a full-pitch resolution of 655 nm at a wavelength of 525 nm (effective NA = 0.8) across a wide field-of-view (FOV) of 1.77 mm2, corresponding to a space–bandwidth–time product of 411 megapixels per second. Our work reveals an important capability of FPM towards high-speed high-throughput imaging of in vitro live cells, achieving video-rate QPI performance across a wide range of scales, both spatial and temporal.

As a typical computational method, Fourier ptychographic microscopy (FPM) can realize high spatial resolution and quantitative phase imaging while preserving the large field of view with a low numerical aperture (NA) objective. A programmable light-emitting diode (LED) array is used as a typical illuminator in an FPM system, and the illumination parameters of each LED element are crucial to the success of the FPM reconstruction algorithm. Compared with LED arrays arranged in rectangular arrays, LED arrays with special structures such as domes or rings can effectively improve FPM imaging results and imaging efficiency. As a trade-off, their calibration difficulty is greatly increased due to the lack of geometric constraints of rectangular arrays. In this paper, we propose an effective hybrid full-pose parameter calibration method for freeform LED array illuminators, combining stereoscopic 3D imaging techniques and the geometric constraints of the microscopic platform. First, a stereovision system is used to obtain the accurate 3D position of each LED element of the freeform illuminator and to construct a rigid 3D coordinate LED array system. Then, calibration between the coordinate system of the LED array and that of the optical imaging component is realized according to the geometric features of the brightfield-to-darkfield edges. Finally, we verify the feasibility and effectiveness of the proposed method through full-pose parameter calibration of LED arrays with different arrangement rules.

Adaptive denoising method for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a newly developed microscopic technique for large field of view, high-resolution and quantitative phase imaging by combining the techniques from ptychographic imaging, aperture synthesizing and phase retrieval. In FPM, an LED array is utilized to illuminate the specimen from different angles and the corresponding intensity images are synthesized to reconstruct a high-resolution complex field. As a flexible and low-cost approach to achieve high-resolution, wide-field and quantitative phase imaging, FPM is of enormous potential in biomedical applications such as hematology and pathology. Conventionally, the FPM reconstruction problem is solved by using a phase retrieval method, termed Alternate Projection. By iteratively updating the Fourier spectrum with low-resolution-intensity images, the result converges to a high-resolution complex field. Here we propose a new FPM reconstruction framework with deep learning methods and design a multiscale, deep residual neural network for FPM reconstruction. We employ the widely used open-source deep learning library PyTorch to train and test our model and carefully choose the hyperparameters of our model. To train and analyze our model, we build a large-scale simulation dataset with an FPM imaging model and an actual dataset captured with an FPM system. The simulation dataset and actual dataset are separated as training datasets and test datasets, respectively. Our model is trained with the simulation training dataset and fine tuned with the fine-tune dataset, which contains actual training data. Both our model and the conventional method are tested on the simulation test dataset and the actual test dataset to evaluate the performances. We also show the reconstruction result of another neural network-based method for comparison. The experiments demonstrate that our model achieves better reconstruction results and consumes much less time than conventional methods. The results also point out that our model is more robust under system aberrations such as noise and blurring (fewer intensity images) compared with conventional methods.

Fourier ptychography microscopy(FPM) is a recently developed computational imaging approach for microscopic super-resolution imaging. By turning on each light-emitting-diode (LED) located on different position on the LED array sequentially and acquiring the corresponding images that contain different spatial frequency components, high spatial resolution and quantitative phase imaging can be achieved in the case of large field-of-view. Nevertheless, FPM has high requirements for the system construction and data acquisition processes, such as precise LEDs position, accurate focusing and appropriate exposure time, which brings many limitations to its practical applications. In this paper, inspired by artificial neural network, we propose a Fourier ptychography multi-parameter neural network (FPMN) with composite physical prior optimization. A hybrid parameter determination strategy combining physical imaging model and data-driven network training is proposed to recover the multi layers of the network corresponding to different physical parameters, including sample complex function, system pupil function, defocus distance, LED array position deviation and illumination intensity fluctuation, etc. Among these parameters, LED array position deviation is recovered based on the features of brightfield to darkfield transition low-resolution images while the others are recovered in the process of training of the neural network. The feasibility and effectiveness of FPMN are verified through simulations and actual experiments. Therefore FPMN can evidently reduce the requirement for practical applications of FPM.

White blood cells (WBC) play a significant role in human immune system, so WBC detection is a meaningful work. In this paper, we propose a novel framework which combines Fourier ptychographic microscopy (FPM) and SO-you only look once (YOLO) for WCB detection. FPM is a recently developed microscope technology which can achieve high-resolution, wide-field, and quantitative phase imaging at the same time without mechanical moving and phase measurement. With the FPM, we can get high resolution, wide field-of-view blood cell images at one time. After obtaining high-resolution images, we propose SO-YOLO as our detection method. As a convolutional neural network, the SO-YOLO outperforms the state-of-the-art detection methods both in accuracy and speed. The proposed framework in this paper can be used in clinical diagnose, because the experiment setup is flexible and the detection is fast. The data set in this paper is made with FPM by ourselves. Our data set contains 1000 color high resolution images and all the WBC has been detected and counted by a human expert. The detection results by the experts are regarded as the ground truth. The experiment results show that the SO-YOLO performs well in detecting small objects compared with other methods.

We propose a physics-assisted deep neural network scheme in Fourier ptychographic microscopy (FPM) using untrained deep neural network priors (FPMUP) to achieve a high-resolution image reconstruction from multiple low-resolution images. Unlike the traditional training type of deep neural network that requires a large labelled dataset, this proposed scheme does not require training and instead outputs the high-resolution image by optimizing the parameters of neural networks to fit the experimentally measured low-resolution images. Besides the amplitude and phase of the sample function, another two parallel neural networks that generate the general pupil function and illumination intensity factors are incorporated into the carefully designed neural networks, which effectively improves the image quality and robustness when both the aberration and illumination intensity fluctuation are present in FPM. Reconstructions using simulated and experimental datasets are demonstrated, showing that the FPMUP scheme has better image quality than the traditional iterative algorithms, especially for the phase recovery, but at the expense of increasing computational cost. Most importantly, it is found that the FPMUP scheme can predict the Fourier spectrum of the sample outside synthetic aperture of FPM and thus eliminate the ringing effect of the recovered images due to the spectral truncation. Inspired by deep image prior in the field of image processing, we may impute the expansion of Fourier spectrums to the deep prior rooted in the architecture of the careful designed four parallel deep neural networks. We envisage that the resolution of FPM will be further enhanced if the Fourier spectrum of the sample outside the synthetic aperture of FPM is accurately predicted.

Fourier ptychography microscopy(FPM) is a recently developed computational imaging approach for microscopic super-resolution imaging. Nevertheless, FPM has high requirements for the system construction and data acquisition processes which brings many limitations to its practical applications. In this paper, we propose a Fourier ptychography multi-parameter neural network (FPMN) with composite physical prior optimization. A hybrid parameter determination strategy combining physical imaging model and data-driven network training is proposed to recover the multi layers of the network corresponding to different physical parameters, including sample complex function, system pupil function, defocus distance, LED array position deviation and illumination intensity fluctuation, etc. Among these parameters, LED array position deviation is recovered based on the features of brightfield to darkfield transition low-resolution images while the others are recovered in the process of training of the neural network. The feasibility and effectiveness of FPMN are verified through simulations and actual experiments. Therefore, FPMN can evidently reduce the requirement for practical applications of FPM.

Fourier ptychographic microscopy (FPM) is an attractive method to extend the resolution beyond the conventional limit defined by a microscope optics, sharing properties with ptychographic, synthetic aperture imaging and phase retrieval. The algorithm uses a sequence of low-resolution (LR) images acquired under angularly varying illumination to reconstruct a high-resolution (HR) image. However, traditional FPM may trap to the sub-optimal solution, since brute-force replacement in the Fourier domain is applied. To address this problem, we propose here a weighted replacement for Fourier ptychographic microscopy (WFPM). We employ the weighted average of spectrums corresponding to different illumination angle to replace the overlapped regions in the Fourier domain. A series of experimental results demonstrate that the reconstructed image using the proposed WFPM shows a better quality and a faster convergence compared with the results obtained by FPM.

Pushing the resolution limit to the nanoscale is a critical challenge for applying the reflective Fourier ptychographic microscopy (FPM) to metrologies for characterization of nanoscale features. Characterization of opaque nanoscale samples using reflective FPM requires chiefly a light source with shorter wavelength to obtain nanoscale resolution, as state-of-the-art device sizes have become sub-100 nm or deep sub-wavelength. We report a reflective deep ultraviolet (DUV) FPM featured by an aperture scanning illumination based on the epi-illumination scheme for accommodating a 193 nm excimer laser source and a high numerical aperture (NA) catadioptric objective lens. The illumination system enables robust control of high-NA angular illumination and optimal energy fluence for FPM imaging and prevention of damage to the sample and optical components. The implemented reflective DUV FPM demonstrated image reconstruction of multiline targets with a minimum linewidth of 80 nm with an average contrast six times higher than conventional DUV microscopy.

Image super-resolution is of great interest in medical imaging field. However, different from natural images studied in computer vision field, the low-resolution (LR) medical imaging data is often a stack of high-resolution (HR) 2D slices with large slice thickness. Consequently, the goal of super-resolution for medical imaging data is to reconstruct the missing slice(s) between any two consecutive slices. Since some modalities (e.g., T1-weighted MR image) are often acquired with high-resolution (HR) image, it is intuitive to harness the prior self-similarity information in the HR image for guiding the super-resolution of LR image (e.g., T2-weighted MR image). The conventional way is to find the profile of patchwise self-similarity in the HR image and then use it to reconstruct the missing information at the same location of LR image. However, the local morphological patterns could vary significantly across the LR and HR images, due to the use of different imaging protocols. Therefore, such direct (un-supervised) adaption of self-similarity profile from HR image is often not effective in revealing the actual information in the LR image. To this end, we propose to employ the existing image information in the LR image to supervise the estimation of self-similarity profile by requiring it not only being optimal in representing patches in the HR image, but also producing less reconstruction errors for the existing image information in the LR image. Moreover, to make the anatomical structures spatially consistent in the reconstructed image, we simultaneously estimate the self-similarity profiles for a stack of patches across consecutive slices by solving a group sparse patch representation problem. We have evaluated our proposed super-resolution method on both simulated brain MR images and real patient images with multiple sclerosis lesion, achieving promising results with more anatomical details and sharpness.

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

As the core task of the reconstruction in conventional ptychography (CP) and Fourier ptychographic microscopy (FPM), the meticulous design of ptychographical iterative engine (PIE) largely affects the performance of reconstruction algorithms. Compared to traditional PIE algorithms, the paradigm of combining with machine learning to cross a local optimum has recently achieved significant progress. Nevertheless, existing designed engines still suffer drawbacks such as excessive hyper-parameters, heavy tuning work and lack of compatibility, which greatly limit their practical applications. In this work, we present a complete set of alternative schemes comprised of a kind of new perspective, a uniform design template, and a fusion framework, to naturally integrate Fourier ptychography (FP) with machine learning concepts. The new perspective, Dynamic Physics, is taken as the preferred tool to analyze a path (algorithm) at the physical level; the uniform design template, T-FP, clarifies the physical significance and optimization part in a path; the fusion framework follows two workable guidelines that are specially designed to keep convergence and make later localized modification for a new path, and further establishes a link between FP iterations and the gradient update in machine learning. Our scheme is compatible with both traditional FP paths and machine learning concepts. By combining ideas in both fields, we offer two design examples, MaFP and AdamFP. Results for both simulations and experiments show that designed algorithms following our scheme obtain better, faster (converge at the early stage after a few iterations) and more stable recovery with only minimal tuning hyper-parameters, demonstrating the effectiveness and superiority of our scheme.

Structural imaging using light microscopy is a cornerstone of histology and cytology. However, the utility of the optical microscope for diagnostic imaging is limited by the fundamental tradeoff between the field of view and spatial resolution and a reliance on exogenous dyes to generate sufficient image contrast. Fourier Ptychographic Microscopy (FPM) is a complex imaging modality with the potential to overcome these limitations by recovering high-resolution images of sample amplitude and phase from a set of low-resolution raw images captured under inclined illumination. In this article we explore the application of FPM to clinical imaging using a simple, low-cost FPM system and simulated and experimental data to explore the influence of both image acquisition parameters and hardware configuration on image quality and imaging throughput. The practical performance of the method is investigated by imaging peripheral blood films and histological tissue sections. We find that, at the cost of increased computational complexity, FPM increases the information capture capacity of the optical microscope significantly, allowing label-free examination and quantification of features such as tissue and cell morphology over large sample areas.

Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. Acquiring color images of histology slides often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learning approach for performing unsupervised image-to-image translation of FPM reconstructions. A cycle-consistent adversarial network with multiscale structure similarity loss is trained to perform virtual brightfield and fluorescence staining of the recovered FPM images. In the training stage, we feed the network with two sets of unpaired images: (1) monochromatic FPM recovery and (2) color or fluorescence images captured using a regular microscope. In the inference stage, the network takes the FPM input and outputs a virtually stained image with reduced coherent artifacts and improved image quality. We test the approach on various samples with different staining protocols. High-quality color and fluorescence reconstructions validate its effectiveness.