Deep neural network based on multi-level wavelet and attention for structured illumination microscopy

The objective of optical super-resolution imaging is to acquire reliable sub-diffraction information on bioprocesses to facilitate scientific discovery. Structured illumination microscopy (SIM) is acknowledged as the optimal modality for live-cell super-resolution imaging. Although recent deep learning techniques have substantially advanced SIM, their transparency and reliability remain uncertain and under-explored, often resulting in unreliable results and biological misinterpretation. Here, we develop Bayesian deep learning (BayesDL) for SIM, which enhances the reconstruction of densely labeled structures while enabling the quantification of super-resolution uncertainty. With the uncertainty, BayesDL-SIM achieves high-fidelity distribution-informed SIM imaging, allowing for the communication of credibility estimates to users regarding the model outcomes. We also demonstrate that BayesDL-SIM boosts SIM reliability by identifying and preventing erroneous generalizations in various model misuse scenarios. Moreover, the BayesDL uncertainty shows versatile utilities for daily super-resolution imaging, such as error estimation, data acquisition evaluation, etc. Furthermore, we demonstrate the effectiveness and superiority of BayesDL-SIM in live-cell imaging, which reliably reveals F-actin dynamics and the reorganization of the cell cytoskeleton. This work lays the foundation for the reliable implementation of deep learning-based SIM methods in practical applications.

Structured illumination microscopy (SIM) is one of the most powerful and versatile super-resolution methods due to its live-cell imaging ability of subcellular structures with high speed. In this paper, we propose an alternative SIM assisted by Bloch surface wave (BSW). Through replacing the conventional laser interference fringes with sub-diffraction BSW counterparts with higher spatial frequency, the imaging resolution of SIM could be enhanced greatly and the super-resolution imaging capability down to 80 nm could be achieved. Compared with traditional wide-field fluorescence microscopy, this BSW SIM demonstrates 2.78 times enhancement in imaging resolution, which surpasses general SIM with only 2 times improvement. Moreover, the structured illumination intensity could be boosted drastically, which is beneficial to nonlinear super-resolution imaging techniques, such as saturated SIM. This BSW SIM would provide a super-resolution, wide field of view approach for surface super-resolution fluorescent imaging while maintaining high imaging speed and good bio-compatibility.

Most current superresolution (SR) microscope techniques surpass the diffraction limit at the expense of temporal resolution, compromising their applications to live-cell imaging. Here we describe a new SR fluorescence microscope based on confocal microscope optics, which we name the spinning disk superresolution microscope (SDSRM). Theoretically, the SDSRM is equivalent to a structured illumination microscope (SIM) and achieves a spatial resolution of 120 nm, double that of the diffraction limit of wide-field fluorescence microscopy. However, the SDSRM is 10 times faster than a conventional SIM because SR signals are recovered by optical demodulation through the stripe pattern of the disk. Therefore a single SR image requires only a single averaged image through the rotating disk. On the basis of this theory, we modified a commercial spinning disk confocal microscope. The improved resolution around 120 nm was confirmed with biological samples. The rapid dynamics of micro-tubules, mitochondria, lysosomes, and endosomes were observed with temporal resolutions of 30-100 frames/s. Because our method requires only small optical modifications, it will enable an easy upgrade from an existing spinning disk confocal to a SR microscope for live-cell imaging.

Structured illumination microscopy (SIM) is one of the most significant widefield super-resolution optical imaging techniques. The conventional SIM utilizes a sinusoidal structured pattern to excite the fluorescent sample; which eventually down-modulates higher spatial frequency sample information within the diffraction-limited passband of the microscopy system and provides around two-fold resolution enhancement over diffraction limit after suitable computational post-processing. Here we provide an overview of the basic principle, image reconstruction, technical development of the SIM technique. Nonetheless, in order to push the SIM resolution further towards the extreme nanoscale dimensions, several different approaches are launched apart from the conventional SIM. Among the various SIM methods, some of the important techniques e.g. TIRF, non-linear, plasmonic, speckle SIM etc are discussed elaborately. Moreover, we highlight different implementations of SIM in various other imaging modalities to enhance their imaging performances with augmented capabilities. Finally, some future outlooks are mentioned which might develop fruitfully and pave the way for new discoveries in near future.

Structured-illumination microscopy (SIM) offers a twofold resolution enhancement beyond the optical diffraction limit. At present, SIM requires several raw structured-illumination (SI) frames to reconstruct a super-resolution (SR) image, especially the time-consuming reconstruction of speckle SIM, which requires hundreds of SI frames. Considering this, we herein propose an untrained structured-illumination reconstruction neural network (USRNN) with known illumination patterns to reduce the amount of raw data that is required for speckle SIM reconstruction by 20 times and thus improve its temporal resolution. Benefiting from the unsupervised optimizing strategy and CNNs' structure priors, the high-frequency information is obtained from the network without the requirement of datasets; as a result, a high-fidelity SR image with approximately twofold resolution enhancement can be reconstructed using five frames or less. Experiments on reconstructing non-biological and biological samples demonstrate the high-speed and high-universality capabilities of our method.

Structured illumination microscopy (SIM) is an attractive choice for fast super-resolution imaging. The generation of structured illumination patterns made by interference of laser beams is broadly employed to obtain high modulation depth of patterns, while the polarizations of the laser beams must be elaborately controlled to guarantee the high contrast of interference intensity, which brings a more complex configuration for the polarization control. The emerging pattern projection strategy is much more compact, but the modulation depth of patterns is deteriorated by the optical transfer function (OTF) of the optical system, especially in high spatial frequency near diffraction limit. Therefore, the traditional super-resolution reconstruction algorithm for interference-based SIM will suffer from much artifact in the case of projection-based SIM that possesses a low modulation depth. Here, we propose an alternative reconstruction algorithm based on image recombination transform (IRT), which provides an alternative solution to address this problem even in a weak modulation depth. We demonstrated the effectiveness of this algorithm in the multicolor super-resolution imaging of BPAE cells in our developed projection-based SIM system, which applies a computer controlled digital micro-mirror device (DMD) for fast fringe generation and multicolor LEDs for illumination. The merit of the system incorporated with the proposed algorithm allows for a low excitation intensity fluorescence imaging even less than 1W/cm2, which is beneficial for the long-term, in vivo super-resolved imaging of live cells and tissues.

Structured illumination microscopy (SIM) is an important super-resolution-based microscopy technique that breaks the diffraction limit and enhances optical microscopy systems. With the development of biology and medical engineering, there is a high demand for real-time and robust SIM imaging under extreme low-light and short-exposure environments. Existing SIM techniques typically require multiple structured illumination frames to produce a high-resolution image. In this article, we propose a single-frame SIM (SF-SIM) based on deep learning. Our SF-SIM only needs one shot of a structured illumination frame and generates similar results compared with the traditional SIM systems that typically require 15 shots. In our SF-SIM, we propose a noise estimator that can effectively suppress the noise in the image and enable our method to work in the low-light and short-exposure environment without the need for stacking multiple frames for nonlocal denoising. We also design a bandpass attention module that makes our deep network more sensitive to the change of frequency and enhances the imaging quality. Our proposed SF-SIM is almost 14 times faster than traditional SIM methods when achieving similar results. Therefore, our method is significantly valuable for the development of microbiology and medicine.

Structured illumination microscopy (SIM) is a fast and gentle super-resolution fluorescence imaging technique, featuring live-cell compatible excitation light levels and high imaging speeds. To achieve SIM, spatial modulation of the fluorescence excitation light is employed. This is typically achieved by interfering coherent laser beams in the sample plane, which are often created by spatial light modulators (SLMs). Digital micromirror devices (DMDs) are a form of SLMs with certain advantages, such as high speed, low cost and wide availability, which present certain hurdles in their implementation, mainly the blazed grating effect caused by the jagged surface structure of the tilted mirrors. Recent works have studied this effect through modelling, simulations and experiments, and laid out possible implementations of multi-color SIM imaging based on DMDs. Here, we present an implementation of a dual-color DMD based SIM microscope using temperature-controlled wavelength matching. By carefully controlling the output wavelength of a diode laser by temperature, we can tune two laser wavelengths in such a way that no opto-mechanical realignment of the SIM setup is necessary when switching between both wavelengths. This reduces system complexity and increases imaging speed. With measurements on nano-bead reference samples, as well as the actin skeleton and membrane of fixed U2OS cells, we demonstrate the capabilities of the setup.

Super-resolution structured illumination microscopy (SR-SIM) is one of the important techniques that are most suitable for live-cell imaging. The reconstructed SR-SIM images are noisy once the raw images are recorded with low-light exposure. Here, we propose a new network (entitled the ND-SIM network) to denoise the SR images reconstructed using frequency-domain algorithms (FDAs). We demonstrate that ND-SIM can yield artifact-free SR images using raw images with an average photon count down to 20 per pixel while achieving comparable resolution to the ground truth (GT) obtained with high-light exposure. We can envisage that the ND-SIM will be widely applied for the long-term, super-resolution live-cell imaging of various bioprocesses in the future.

Biological images captured by microscopes are characterized by heterogeneous signal-to-noise ratios (SNRs) due to spatially varying photon emission across the field of view convoluted with camera noise. State-of-the-art unsupervised structured illumination microscopy (SIM) reconstruction methods, commonly implemented in the Fourier domain, often do not accurately model this noise. Such methods therefore suffer from high-frequency artifacts, user-dependent choices of smoothness constraints making assumptions on biological features, and unphysical negative values in the recovered fluorescence intensity map. On the other hand, supervised algorithms rely on large datasets for training, and often require retraining for new sample structures. Consequently, achieving high contrast near the maximum theoretical resolution in an unsupervised, physically principled manner remains an open problem. Here, we propose Bayesian-SIM (B-SIM), a Bayesian framework to quantitatively reconstruct SIM data, rectifying these shortcomings by accurately incorporating known noise sources in the spatial domain. To accelerate the reconstruction process, we use the finite extent of the point-spread-function to devise a parallelized Monte Carlo strategy involving chunking and restitching of the inferred fluorescence intensity. We benchmark our framework on both simulated and experimental images, and demonstrate improved contrast permitting feature recovery at up to 25% shorter length scales over state-of-the-art methods at both high- and low SNR. B-SIM enables unsupervised, quantitative, physically accurate reconstruction without the need for labeled training data, democratizing high-quality SIM reconstruction and expands the capabilities of live-cell SIM to lower SNR, potentially revealing biological features in previously inaccessible regimes.

The super-resolution imaging technique of structured illumination microscopy (SIM) enables the mixing of high-frequency information into the optical transmission domain via light-source modulation, thus breaking the optical diffraction limit. Correlative SIM, which combines other techniques with SIM, offers more versatility or higher imaging resolution than traditional SIM. In this review, we first briefly introduce the imaging mechanism and development trends of conventional SIM. Then, the principles and recent developments of correlative SIM techniques are reviewed. Finally, the future development directions of SIM and its correlative microscopies are presented.

We present and demonstrate an efficient method for the reconstruction of profiles acquired by multifocal structured illumination microscopy (MSIM) utilizing few raw images. Firstly, we propose a method to produce nine raw multifocal images with enhanced modulation depth to accomplish the uniform illumination of the sample. Then, combing with the parameter of the arrays, we perform the standard construct reconstruction procedure of structured illumination microscopy (SIM) row by row and column by column. Finally, we combine these restored images together to obtain the final image with enhanced resolution and good contrast. Based on theoretical analysis and numerical simulations, this method shows great potential in the field of the image reconstruction of MSIM data.

Nonlinear structured illumination microscopy (SIM) allows full-field imaging at resolutions <100 nm. Two nonlinear effects, excitation saturation (SSIM) and the photo-switching of protein had been applied to nonlinear SIM. We report a new SIM technique which utilizes the nonlinearity of STED effect. Resolution and signal noise ratio simulation shows that STED-SIM may serve as a better alternative to SSIM and SIM with photo-switchable protein. SIM requires a strong nonlinear effect in a large area. We use Surface Plasmon Resonant to enhance of evanescence field near a dielectric-metal-dielectric interface. An 8 times STED effect enhancement is achieved on an optimized glass-silver-glass-water planar structure. We further use the interference of two SPR-enhanced STED fields propagating at opposite direction to generate a 1D structured STED field. Combined with a uniform excitation field, the structure STED field allows full field total internal reflection imaging with an enhanced resolution along the structured dimension. Less than 50 nm resolution is demonstrated. A STED-SIM microscope with 2D structured STED field is under development. Future research will apply the microscope to superresolution imaging of membrane resident or near membrane structure at super-resolution in live cells.

Abstract: As an early component of modern super-resolution (SR) imaging technology, structured illumination microscopy (SIM) has been developed for nearly twenty years. With up to ~60 nm and 564 Hz frame rate, it has achieved an optimal combination of spatiotemporal resolution in live cells recently. Despite these advantages, SIM also suffers disadvantages, some of which originated from the intrinsic reconstruction process. Here we review recent technical advances in SIM, including SR reconstruction, performance evaluation, and its integration with other technologies to provide a practical guide for biologists.

Structured illumination microscopy (SIM) achieves superresolution in fluorescence imaging through patterned illumination and computational image reconstruction, yet current methods require bulky, costly modulation optics and high-precision optical alignment, thus hindering the widespread implementation of SIM. To address this challenge, this work demonstrates how nano-optical metasurfaces, rationally designed to tailor the far-field optical wavefront at sub-wavelength dimensions, hold great potential as ultrathin, single-surface, all-optical wavefront modulators for SIM. We computationally demonstrate this principle with a multipolar-resonant metasurface composed of silicon nanostructures that generate versatile optical wavefronts in the far field upon variation of the polarization or angle of incident light. Algorithmic optimization is performed to identify the seven most suitable illumination patterns for SIM generated by the metasurface based on three key criteria. We quantitatively demonstrate that multipolar-resonant metasurface SIM (mrm-SIM) achieves resolution gain that is comparable to conventional methods by applying the seven optimal metasurface-generated wavefronts to simulated fluorescent objects and reconstructing the objects using proximal gradient descent. Notably, we show that mrm-SIM achieves these resolution gains with a far-field illumination pattern that circumvents complex equipment and alignment requirements of comparable methodologies. The work presented here paves the way for a metasurface-enabled experimental simplification of structured illumination microscopy.