Fast structured illumination microscopy using rolling shutter cameras

Modern automotive vehicles are often equipped with a budget commercial rolling shutter (RS) camera. These devices usually produce distorted images due to the camera's inter-row delay while capturing the image. They will defeat the following computer vision tasks, such as ego-motion estimation. In this paper, we present a closed-form solver for a vehicle-mounted RS camera ego-motion estimation problem with only one point-match between two consecutive distorted RS images, called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RS-1 pt</b>. To the best of our knowledge, this is the first work to address the ego-motion estimation problem for vehicle-mounted RS cameras specifically. Thanks to the 1 pt minimal solver, extensive experiments demonstrate the proposed method surpasses classical global shutter based (GS-based) solvers and existing RS-based solvers by a significant margin in terms of both efficiency and effectiveness. Besides, we synthesize and self-collect three vehicle-mounted RS camera datasets with ground-truth GS images and trajectory, which are absent from the existing works and will be publicly available as a benchmark to the research community.

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

Structured illumination microscopy (SIM) is a fast and gentle super-resolution fluorescence imaging technique, featuring live-cell compatible excitation light levels and high imaging speeds. To achieve SIM, spatial modulation of the fluorescence excitation light is employed. This is typically achieved by interfering coherent laser beams in the sample plane, which are often created by spatial light modulators (SLMs). Digital micromirror devices (DMDs) are a form of SLMs with certain advantages, such as high speed, low cost and wide availability, which present certain hurdles in their implementation, mainly the blazed grating effect caused by the jagged surface structure of the tilted mirrors. Recent works have studied this effect through modelling, simulations and experiments, and laid out possible implementations of multi-color SIM imaging based on DMDs. Here, we present an implementation of a dual-color DMD based SIM microscope using temperature-controlled wavelength matching. By carefully controlling the output wavelength of a diode laser by temperature, we can tune two laser wavelengths in such a way that no opto-mechanical realignment of the SIM setup is necessary when switching between both wavelengths. This reduces system complexity and increases imaging speed. With measurements on nano-bead reference samples, as well as the actin skeleton and membrane of fixed U2OS cells, we demonstrate the capabilities of the setup.

Structured illumination microscopy (SIM) is an important super-resolution-based microscopy technique that breaks the diffraction limit and enhances optical microscopy systems. With the development of biology and medical engineering, there is a high demand for real-time and robust SIM imaging under extreme low-light and short-exposure environments. Existing SIM techniques typically require multiple structured illumination frames to produce a high-resolution image. In this article, we propose a single-frame SIM (SF-SIM) based on deep learning. Our SF-SIM only needs one shot of a structured illumination frame and generates similar results compared with the traditional SIM systems that typically require 15 shots. In our SF-SIM, we propose a noise estimator that can effectively suppress the noise in the image and enable our method to work in the low-light and short-exposure environment without the need for stacking multiple frames for nonlocal denoising. We also design a bandpass attention module that makes our deep network more sensitive to the change of frequency and enhances the imaging quality. Our proposed SF-SIM is almost 14 times faster than traditional SIM methods when achieving similar results. Therefore, our method is significantly valuable for the development of microbiology and medicine.

Traditionally, camera-based tracking approaches have treated rolling shutter and radial distortion as imaging artifacts that have to be overcome and corrected for in order to apply standard camera models and scene reconstruction methods. In this paper, we introduce a novel multi-camera tracking approach that for the first time jointly leverages the information introduced by rolling shutter and radial distortion as a feature to achieve superior performance with respect to high-frequency camera pose estimation. In particular, our system is capable of attaining high tracking rates that were previously unachievable. Our approach explicitly leverages rolling shutter capture and radial distortion to process individual rows, rather than entire image frames, for accurate camera motion estimation. We estimate a per-row 6 DoF pose of a rolling shutter camera by tracking multiple points on a radially distorted row whose rays span a curved surface in 3D space. Although tracking systems for rolling shutter cameras exist, we are the first to leverage radial distortion to measure a per-row pose - enabling us to use less than half the number of cameras required by the previous state of the art. We validate our system on both synthetic and real imagery.

Structured illumination has emerged as a pivotal technique in biomedical imaging and machine vision, driving breakthroughs in optical sectioning, super-resolution microscopy, phase imaging, and 3D surface profiling. These applications rely on precise light patterns projected onto samples to extract high-resolution information, where spatial light modulators (SLMs) are widely used for their high speed, precision, and flexibility. However, the limited pixel count of SLMs constrains the spatial bandwidth product (SBP) of structured illumination, creating a tradeoff between the field-of-view (FoV) and spatial resolution. To address this limitation, we develop a mesoscale structured illumination technique that integrates a custom-designed pyramidal lens with an SLM; this technique achieves an unprecedented SBP enhancement of over 100-fold. We demonstrate the generation of mesoscale structured illumination fields containing more than 5000 stripes with a 6-step phase shift using only a 1024-pixel SLM. Leveraging this approach, we further develop a multicolor super-resolution structured illumination microscopy PyramidalSIM, which delivers a reconstructed FoV of 4K pixels and doubled spatial resolution yet utilizes only 609 SLM pixels. Our approach effectively surpasses the intrinsic SBP limitation of SLMs, paving the way for applications such as high-throughput super-resolution imaging and large-scale surface metrology.

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Most consumer cameras use rolling shutter (RS) exposure, the captured videos often suffer from distortions (e.g., skew and jelly effect). Also, these videos are impeded by the limited bandwidth and frame rate, which inevitably affect the video streaming experience. In this paper, we excavate the potential of event cameras as they enjoy high temporal resolution. Accordingly, we propose a framework to recover the global shutter (GS) high frame rate (i.e., slow motion) video without RS distortion from an RS camera and event camera. One challenge is the lack of real-world datasets for supervised training. Therefore, we explore self-supervised learning with the key idea of estimating the displacement field-a non-linear and dense 3D spatiotemporal representation of all pixels during the exposure time. This allows for a mutual reconstruction between RS and GS frames and facilitates slow-motion video recovery. We then combine the input RS frames with the DF to map them to the GS frames (RS-to-GS). Given the under-constrained nature of this mapping, we integrate it with the inverse mapping (GS-to-RS) and RS frame warping (RS-to-RS) for self-supervision. We evaluate our framework via objective analysis (i.e., quantitative and qualitative comparisons on four datasets) and subjective studies (i.e., user study). The results show that our framework can recover slow-motion videos without distortion, with much lower bandwidth (94% drop) and higher inference speed ($ 16\; {\rm ms}/{\rm frame}$16 ms / frame ) under $32 \times$32× frame interpolation.

Digital micromirror devices (DMDs) are spatial light modulators that employ the electro-mechanical movement of miniaturized mirrors to steer and thus modulate the light reflected off a mirror array. Their wide availability, low cost and high speed make them a popular choice both in consumer electronics such as video projectors, and scientific applications such as microscopy. High-end fluorescence microscopy systems typically employ laser light sources, which by their nature provide coherent excitation light. In super-resolution microscopy applications that use light modulation, most notably structured illumination microscopy (SIM), the coherent nature of the excitation light becomes a requirement to achieve optimal interference pattern contrast. The universal combination of DMDs and coherent light sources, especially when working with multiple different wavelengths, is unfortunately not straight forward. The substructure of the tilted micromirror array gives rise to a blazed grating, which has to be understood and which must be taken into account when designing a DMD-based illumination system. Here, we present a set of simulation frameworks that explore the use of DMDs in conjunction with coherent light sources, motivated by their application in SIM, but which are generalizable to other light patterning applications. This framework provides all the tools to explore and compute DMD-based diffraction effects and to simulate possible system alignment configurations computationally, which simplifies the system design process and provides guidance for setting up DMD-based microscopes.This article is part of the Theo Murphy meeting ‘Super-resolution structured illumination microscopy (part 1)’.

We developed a structured illumination microscopy (SIM) system that uses a spatial light modulator (SLM) to generate interference illumination patterns at four orientations - 0 degrees, 45 degrees, 90 degrees, and 135 degrees, to reconstruct a high-resolution image. The use of a SLM for pattern alterations is rapid and precise, without mechanical calibration; moreover, our design of SLM patterns allows generating the four illumination patterns of high contrast and nearly equivalent periods to achieve a near isotropic enhancement in lateral resolution. We compare the conventional image of 100-nm beads with those reconstructed from two (0 degrees +90 degrees or 45 degrees +135 degrees) and four (0 degrees +45 degrees +90 degrees +135 degrees) pattern orientations to show the differences in resolution and image, with the support of simulations. The reconstructed images of 200-nm beads at various depths and fine structures of actin filaments near the edge of a HeLa cell are presented to demonstrate the intensity distributions in the axial direction and the prospective application to biological systems.

Linear structured illumination microscopy (SIM) is a super-resolution microscopy technique that does not impose photophysics requirements on fluorescent samples. Multicolor SIM implementations typically rely on liquid crystal on silicon (LCoS) spatial light modulators (SLM's) for patterning the excitation light, but digital micromirror devices (DMD's) are a promising alternative, owing to their lower cost and higher speed. However, existing coherent DMD SIM implementations use only a single wavelength of light, limited by the lack of efficient approaches for solving the blazed grating effect for polychromatic light. We develop the requisite quantitative tools, including a closed form solution of the blaze and diffraction conditions, forward models of DMD diffraction and pattern projection, and a model of DMD aberrations. Based on these advances, we constructed a three-color DMD microscope, quantified the effect of aberrations from the DMD, developed a high-resolution optical transfer function measurement technique, and demonstrated SIM on fixed and live cells. This opens the door to applying DMD's in polychromatic applications previously restricted to LCoS SLM's.

Cepstrum-based interferometric microscopy (CIM) has risen as a groundbreaking digital holographic microscopy (DHM) technique that overcomes the conventional need of a clean/known reference beam for holographic recording. In CIM, two arbitrary complex object fields interfere each other to provide quantitative phase imaging (QPI) of both at once. Previous CIM implementations were validated using global-shutter (GS) cameras. However, rolling-shutter (RS) cameras provide several advantages, such as reduced price, high frame rate, low electronical noise, and small pixel size in spite of some drawbacks (flickering effect) that are overcome in this Letter. Hereby, we report on the validation of the CIM technique using rolling-shutter cameras including a novel, to the best of our knowledge, flickering correction approach. The performance of the approach is experimentally validated using calibrated phase objects and biological samples.

Structured illumination microscopy (SIM) is one of the most powerful and versatile super-resolution methods due to its live-cell imaging ability of subcellular structures with high speed. In this paper, we propose an alternative SIM assisted by Bloch surface wave (BSW). Through replacing the conventional laser interference fringes with sub-diffraction BSW counterparts with higher spatial frequency, the imaging resolution of SIM could be enhanced greatly and the super-resolution imaging capability down to 80&#x00A0;nm could be achieved. Compared with traditional wide-field fluorescence microscopy, this BSW SIM demonstrates 2.78 times enhancement in imaging resolution, which surpasses general SIM with only 2 times improvement. Moreover, the structured illumination intensity could be boosted drastically, which is beneficial to nonlinear super-resolution imaging techniques, such as saturated SIM. This BSW SIM would provide a super-resolution, wide field of view approach for surface super-resolution fluorescent imaging while maintaining high imaging speed and good bio-compatibility.

Structured illumination microscopy (SIM) is an established technique that allows subdiffraction resolution imaging by heterodyning high sample frequencies into the system's passband via structured illumination. However, until now, SIM has been typically used to achieve subdiffraction resolution for intensity-based imaging. Here, we present a novel optical setup that uses structured illumination with a broadband light source to obtain noise-reduced, subdiffraction resolution, quantitative phase imaging (QPM) of cells. We compare this with a previous work for subdiffraction QPM imaging via SIM that used a laser source, and was thus still corrupted by coherent noise.

Rotating machinery speed extraction through smartphone video acquisition from a radial viewpoint