Multistate Constrained Invariant Kalman Filter For Rolling Shutter Camera And Imu Calibration

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Modern automotive vehicles are often equipped with a budget commercial rolling shutter (RS) camera. These devices usually produce distorted images due to the camera's inter-row delay while capturing the image. They will defeat the following computer vision tasks, such as ego-motion estimation. In this paper, we present a closed-form solver for a vehicle-mounted RS camera ego-motion estimation problem with only one point-match between two consecutive distorted RS images, called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RS-1 pt</b>. To the best of our knowledge, this is the first work to address the ego-motion estimation problem for vehicle-mounted RS cameras specifically. Thanks to the 1 pt minimal solver, extensive experiments demonstrate the proposed method surpasses classical global shutter based (GS-based) solvers and existing RS-based solvers by a significant margin in terms of both efficiency and effectiveness. Besides, we synthesize and self-collect three vehicle-mounted RS camera datasets with ground-truth GS images and trajectory, which are absent from the existing works and will be publicly available as a benchmark to the research community.

Cepstrum-based interferometric microscopy (CIM) has risen as a groundbreaking digital holographic microscopy (DHM) technique that overcomes the conventional need of a clean/known reference beam for holographic recording. In CIM, two arbitrary complex object fields interfere each other to provide quantitative phase imaging (QPI) of both at once. Previous CIM implementations were validated using global-shutter (GS) cameras. However, rolling-shutter (RS) cameras provide several advantages, such as reduced price, high frame rate, low electronical noise, and small pixel size in spite of some drawbacks (flickering effect) that are overcome in this Letter. Hereby, we report on the validation of the CIM technique using rolling-shutter cameras including a novel, to the best of our knowledge, flickering correction approach. The performance of the approach is experimentally validated using calibrated phase objects and biological samples.

Rotating machinery speed extraction through smartphone video acquisition from a radial viewpoint

This paper introduces the asynchronous Optical Camera Communication (asynchronous OCC) using LED and rolling shutter camera as a pair of wireless communication. Aiming to enhance the capacity of unidirectional wireless link, the response of camera is measured, and then the operating frequency range, system architecture, decoding procedure and algorithm are proposed. The frame and super-frame structure are also considered especially for OCC using rolling shutter camera. The performance of implemented system is evaluated and discussed.

A novel automatic video deshearing is proposed to reduce the skew artifacts (jelly artifacts) in sequences captured by CMOS censor cameras. The paper first considers the principles of skew artifacts in rolling shutter cameras. Then a deshearing algorithm with high accuracy is discussed with automatic skew artifact detection. A video completion step is also implemented to reconstruct unavailable pixels in the new grid of the desheared frame. Simulation results show that the proposed method effectively reduces the skew artifacts for different sequences captured by rolling shutter cameras.

Combining cameras and inertial measurement units (IMUs) has been proven effective in motion tracking, as these two sensing modalities offer complementary characteristics that are suitable for fusion. While most works focus on global-shutter cameras and synchronized sensor measurements, consumer-grade devices are mostly equipped with rolling-shutter cameras and suffer from imperfect sensor synchronization. In this work, we propose a nonlinear optimization-based monocular visual inertial odometry (VIO) with varying camera-IMU time offset modeled as an unknown variable. Our approach is able to handle the rolling-shutter effects and imperfect sensor synchronization in a unified way. Additionally, we introduce an efficient algorithm based on dynamic programming and red-black tree to speed up IMU integration over variable-length time intervals during the optimization. An uncertainty-aware initialization is also presented to launch the VIO robustly. Comparisons with state-of-the-art methods on the Euroc dataset and mobile phone data are shown to validate the effectiveness of our approach.

We present DisCo, a novel display-camera communication system. DisCo enables displays and cameras to communicate with each other while also displaying and capturing images for human consumption. Messages are transmitted by temporally modulating the display brightness at high frequencies so that they are imperceptible to humans. Messages are received by a rolling shutter camera that converts the temporally modulated incident light into a spatial flicker pattern. In the captured image, the flicker pattern is superimposed on the pattern shown on the display. The flicker and the display pattern are separated by capturing two images with different exposures. The proposed system performs robustly in challenging real-world situations such as occlusion, variable display size, defocus blur, perspective distortion, and camera rotation. Unlike several existing visible light communication methods, DisCo works with off-the-shelf image sensors. It is compatible with a variety of sources (including displays, single LEDs), as well as reflective surfaces illuminated with light sources. We have built hardware prototypes that demonstrate DisCo’s performance in several scenarios. Because of its robustness, speed, ease of use, and generality, DisCo can be widely deployed in several applications, such as advertising, pairing of displays with cell phones, tagging objects in stores and museums, and indoor navigation.

Image sensors with nondestructive charge readout provide single-photon or single-electron sensitivity, but at the cost of long readout times. We present a smart readout technique to allow the use of these sensors in visible light and other applications that require faster readout times. The method optimizes the readout noise and time by changing the number of times pixels are read out either statically, by defining an arbitrary number of regions of interest in the array, or dynamically, depending on the charge or energy of interest in the pixel. This technique is tested in a Skipper CCD showing that it is possible to obtain deep subelectron noise, and therefore, high resolution of quantized charge, while dynamically changing the readout noise of the sensor. These faster, low noise readout techniques show that the skipper CCD is a competitive technology even where other technologies such as electron multiplier charge coupled devices, silicon photo multipliers, etc. are currently used. This technique could allow skipper CCDs to benefit new astronomical instruments, quantum imaging, exoplanet search and study, and quantum metrology.

Traditionally, camera-based tracking approaches have treated rolling shutter and radial distortion as imaging artifacts that have to be overcome and corrected for in order to apply standard camera models and scene reconstruction methods. In this paper, we introduce a novel multi-camera tracking approach that for the first time jointly leverages the information introduced by rolling shutter and radial distortion as a feature to achieve superior performance with respect to high-frequency camera pose estimation. In particular, our system is capable of attaining high tracking rates that were previously unachievable. Our approach explicitly leverages rolling shutter capture and radial distortion to process individual rows, rather than entire image frames, for accurate camera motion estimation. We estimate a per-row 6 DoF pose of a rolling shutter camera by tracking multiple points on a radially distorted row whose rays span a curved surface in 3D space. Although tracking systems for rolling shutter cameras exist, we are the first to leverage radial distortion to measure a per-row pose - enabling us to use less than half the number of cameras required by the previous state of the art. We validate our system on both synthetic and real imagery.

Spatial light modulators (SLM) update in a synchronous manner, whereas the data readout process in fast structured illumination systems is usually done using a rolling shutter camera with asynchronous readout. In structured illumination microscopy (SIM), this leads to synchronization problems causing a speed limit for fast acquisition. In this paper we present a configuration to overcome this limit by exploiting the extremely fast SLM display and dividing it into several segments along the direction of the rolling shutter of the sCMOS camera and displaying multiple SLM frames per camera acquisition. The sCMOS runs in continuous rolling shutter mode and the SLM keeps the readout-line always inside a dark region presenting different SIM patterns before and after the readout/start-exposure line.Using this approach, we reached a raw frame rate of 714 frames per second (fps) resulting in a two-beam SIM acquisition rate of 79 fps with a region of interest (ROI) of 16.5 × 16.5 μm2.

The conventional calibration methods based on a perspective camera model are not suitable for the underground camera with two-layer glasses, which is specially designed for explosion proof and dust removal in a coal mine. Underground camera modeling and calibration algorithms are urgently needed to improve the precision and reliability of underground visual measurement systems. This article presents a novel geometrically driven underground camera calibration algorithm for a boom-type roadheader. The underground camera model is established under coplanarity constraints, considering explicitly the impact of refraction triggered by the two-layer glasses and deriving the geometrical relationship of equivalent collinearity equations. On this basis, we perform parameters calibration based on a geometrically driven calibration model, which are 2D-2D correspondences between the image points and object coordinates of the planar target. A hybrid Levenberg-Marqurdt (LM) and particle swarm optimization (PSO) algorithm is further proposed in terms of the dynamic combination of the LM and PSO, which optimizes the underground camera calibration results by minimizing the error of the nonlinear underground camera model. The experimental results demonstrate that the pose errors caused by the two-layer glass refraction are well corrected by the proposed method. The accuracy of the cutting-head pose estimation has increased by 55.73%, meeting the requirements of underground excavations.

Many unmanned aerial vehicles (UAV) host rolling shutter (RS) cameras, i.e., cameras where image rows are exposed at slightly different times. As the camera moves in the meantime, this causes inconsistencies in homologous ray intersections in the bundle adjustment, so correction models have been proposed to deal with the problem. This paper presents a series of test flights and simulations performed with different UAV platforms at varying speeds over terrain of various morphologies with the objective of investigating and possibly optimising how RS correction models perform under different conditions, in particular as far as block control is concerned. To this aim, three RS correction models have been applied in various combinations, decreasing the number of fixed ground control points (GCP) or exploiting GNSS-determined camera stations. From the experimental tests as well as from the simulations, four conclusions can be drawn: (a) RS affects primarily horizontal coordinates and varies notably from platform to platform; (b) if the ground control is dense enough, all correction models lead practically to the same mean error on checkpoints; however, some models may cause large errors in elevation if too few GCP are used; (c) in most cases, a specific correction model is not necessary since the affine deformation caused by RS can be adequately modelled by just applying the extended Fraser camera calibration model; (d) using GNSS-assisted block orientation, the number of necessary GCP is strongly reduced.

Most consumer cameras use rolling shutter (RS) exposure, the captured videos often suffer from distortions (e.g., skew and jelly effect). Also, these videos are impeded by the limited bandwidth and frame rate, which inevitably affect the video streaming experience. In this paper, we excavate the potential of event cameras as they enjoy high temporal resolution. Accordingly, we propose a framework to recover the global shutter (GS) high frame rate (i.e., slow motion) video without RS distortion from an RS camera and event camera. One challenge is the lack of real-world datasets for supervised training. Therefore, we explore self-supervised learning with the key idea of estimating the displacement field-a non-linear and dense 3D spatiotemporal representation of all pixels during the exposure time. This allows for a mutual reconstruction between RS and GS frames and facilitates slow-motion video recovery. We then combine the input RS frames with the DF to map them to the GS frames (RS-to-GS). Given the under-constrained nature of this mapping, we integrate it with the inverse mapping (GS-to-RS) and RS frame warping (RS-to-RS) for self-supervision. We evaluate our framework via objective analysis (i.e., quantitative and qualitative comparisons on four datasets) and subjective studies (i.e., user study). The results show that our framework can recover slow-motion videos without distortion, with much lower bandwidth (94% drop) and higher inference speed ($ 16\; {\rm ms}/{\rm frame}$16 ms / frame ) under $32 \times$32× frame interpolation.