Observability Analysis and Keyframe-Based Filtering for Visual Inertial Odometry With Full Self-Calibration

Modern automotive vehicles are often equipped with a budget commercial rolling shutter (RS) camera. These devices usually produce distorted images due to the camera's inter-row delay while capturing the image. They will defeat the following computer vision tasks, such as ego-motion estimation. In this paper, we present a closed-form solver for a vehicle-mounted RS camera ego-motion estimation problem with only one point-match between two consecutive distorted RS images, called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RS-1 pt</b>. To the best of our knowledge, this is the first work to address the ego-motion estimation problem for vehicle-mounted RS cameras specifically. Thanks to the 1 pt minimal solver, extensive experiments demonstrate the proposed method surpasses classical global shutter based (GS-based) solvers and existing RS-based solvers by a significant margin in terms of both efficiency and effectiveness. Besides, we synthesize and self-collect three vehicle-mounted RS camera datasets with ground-truth GS images and trajectory, which are absent from the existing works and will be publicly available as a benchmark to the research community.

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Cepstrum-based interferometric microscopy (CIM) has risen as a groundbreaking digital holographic microscopy (DHM) technique that overcomes the conventional need of a clean/known reference beam for holographic recording. In CIM, two arbitrary complex object fields interfere each other to provide quantitative phase imaging (QPI) of both at once. Previous CIM implementations were validated using global-shutter (GS) cameras. However, rolling-shutter (RS) cameras provide several advantages, such as reduced price, high frame rate, low electronical noise, and small pixel size in spite of some drawbacks (flickering effect) that are overcome in this Letter. Hereby, we report on the validation of the CIM technique using rolling-shutter cameras including a novel, to the best of our knowledge, flickering correction approach. The performance of the approach is experimentally validated using calibrated phase objects and biological samples.

Traditionally, camera-based tracking approaches have treated rolling shutter and radial distortion as imaging artifacts that have to be overcome and corrected for in order to apply standard camera models and scene reconstruction methods. In this paper, we introduce a novel multi-camera tracking approach that for the first time jointly leverages the information introduced by rolling shutter and radial distortion as a feature to achieve superior performance with respect to high-frequency camera pose estimation. In particular, our system is capable of attaining high tracking rates that were previously unachievable. Our approach explicitly leverages rolling shutter capture and radial distortion to process individual rows, rather than entire image frames, for accurate camera motion estimation. We estimate a per-row 6 DoF pose of a rolling shutter camera by tracking multiple points on a radially distorted row whose rays span a curved surface in 3D space. Although tracking systems for rolling shutter cameras exist, we are the first to leverage radial distortion to measure a per-row pose - enabling us to use less than half the number of cameras required by the previous state of the art. We validate our system on both synthetic and real imagery.

This paper introduces the asynchronous Optical Camera Communication (asynchronous OCC) using LED and rolling shutter camera as a pair of wireless communication. Aiming to enhance the capacity of unidirectional wireless link, the response of camera is measured, and then the operating frequency range, system architecture, decoding procedure and algorithm are proposed. The frame and super-frame structure are also considered especially for OCC using rolling shutter camera. The performance of implemented system is evaluated and discussed.

Rotating machinery speed extraction through smartphone video acquisition from a radial viewpoint

Spatial light modulators (SLM) update in a synchronous manner, whereas the data readout process in fast structured illumination systems is usually done using a rolling shutter camera with asynchronous readout. In structured illumination microscopy (SIM), this leads to synchronization problems causing a speed limit for fast acquisition. In this paper we present a configuration to overcome this limit by exploiting the extremely fast SLM display and dividing it into several segments along the direction of the rolling shutter of the sCMOS camera and displaying multiple SLM frames per camera acquisition. The sCMOS runs in continuous rolling shutter mode and the SLM keeps the readout-line always inside a dark region presenting different SIM patterns before and after the readout/start-exposure line.Using this approach, we reached a raw frame rate of 714 frames per second (fps) resulting in a two-beam SIM acquisition rate of 79 fps with a region of interest (ROI) of 16.5 × 16.5 μm2.

Most consumer cameras use rolling shutter (RS) exposure, the captured videos often suffer from distortions (e.g., skew and jelly effect). Also, these videos are impeded by the limited bandwidth and frame rate, which inevitably affect the video streaming experience. In this paper, we excavate the potential of event cameras as they enjoy high temporal resolution. Accordingly, we propose a framework to recover the global shutter (GS) high frame rate (i.e., slow motion) video without RS distortion from an RS camera and event camera. One challenge is the lack of real-world datasets for supervised training. Therefore, we explore self-supervised learning with the key idea of estimating the displacement field-a non-linear and dense 3D spatiotemporal representation of all pixels during the exposure time. This allows for a mutual reconstruction between RS and GS frames and facilitates slow-motion video recovery. We then combine the input RS frames with the DF to map them to the GS frames (RS-to-GS). Given the under-constrained nature of this mapping, we integrate it with the inverse mapping (GS-to-RS) and RS frame warping (RS-to-RS) for self-supervision. We evaluate our framework via objective analysis (i.e., quantitative and qualitative comparisons on four datasets) and subjective studies (i.e., user study). The results show that our framework can recover slow-motion videos without distortion, with much lower bandwidth (94% drop) and higher inference speed ($ 16\; {\rm ms}/{\rm frame}$16 ms / frame ) under $32 \times$32× frame interpolation.

In this paper, we propose a novel algorithm for joint calibration of a rolling shutter camera and inertial sensors. Specifically, the proposed method performs online calibration to estimate the camera intrinsics, the readout time, the extrinsics and the time offset between the camera and the IMU, as well as IMU’s bias. By employing a multi-state constrained invariant Kalman filter, the proposed method processes the measurements of IMU and rolling shutter camera in a decoupled way, which greatly reduces the computational complexity of Jacobian matrices and further improves the robustness and accuracy. Experiments on public datasets demonstrate that the proposed approach can accurately calibrate the considered parameters and perform better than state-of-the-art methods in terms of accuracy and robustness.

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.

Many unmanned aerial vehicles (UAV) host rolling shutter (RS) cameras, i.e., cameras where image rows are exposed at slightly different times. As the camera moves in the meantime, this causes inconsistencies in homologous ray intersections in the bundle adjustment, so correction models have been proposed to deal with the problem. This paper presents a series of test flights and simulations performed with different UAV platforms at varying speeds over terrain of various morphologies with the objective of investigating and possibly optimising how RS correction models perform under different conditions, in particular as far as block control is concerned. To this aim, three RS correction models have been applied in various combinations, decreasing the number of fixed ground control points (GCP) or exploiting GNSS-determined camera stations. From the experimental tests as well as from the simulations, four conclusions can be drawn: (a) RS affects primarily horizontal coordinates and varies notably from platform to platform; (b) if the ground control is dense enough, all correction models lead practically to the same mean error on checkpoints; however, some models may cause large errors in elevation if too few GCP are used; (c) in most cases, a specific correction model is not necessary since the affine deformation caused by RS can be adequately modelled by just applying the extended Fraser camera calibration model; (d) using GNSS-assisted block orientation, the number of necessary GCP is strongly reduced.

A novel automatic video deshearing is proposed to reduce the skew artifacts (jelly artifacts) in sequences captured by CMOS censor cameras. The paper first considers the principles of skew artifacts in rolling shutter cameras. Then a deshearing algorithm with high accuracy is discussed with automatic skew artifact detection. A video completion step is also implemented to reconstruct unavailable pixels in the new grid of the desheared frame. Simulation results show that the proposed method effectively reduces the skew artifacts for different sequences captured by rolling shutter cameras.

Artefacts that are present in images taken from a moving rolling shutter camera degrade the accuracy of absolute pose estimation. To alleviate this problem, we introduce an addition linear velocity in the camera projection matrix to approximate the motion of the rolling shutter camera. In particular, we derive a minimal solution using the Grobner Basis that solves for the absolute pose as well as the motion of a rolling shutter camera. We show that the minimal problem requires 5-point correspondences and gives up to 8 real solutions. We also show that our formulation can be extended to use more than 5-point correspondences. We use RANSAC to robustly get all the inliers. In the final step, we relax the linear velocity assumption and do a non-linear refinement on the fuli motion, i.e. linear and angular velocities, and pose of the rolling shutter camera with all the inliers. We verify the feasibility and accuracy of our algorithm with both simulated and real-world datasets.

Optical camera communication (OCC) provides license-free communication channels without radio waves. The rolling shutter mechanism is expected to enhance the capacity of wireless links in OCC. However, signal erasure occurs due to waiting times between frames in consecutive signal transmission. To address this issue, we propose a symbol timing synchronization for OCC using a rolling shutter camera. The receiver camera synchronizes to the transmitter with additional waiting time which is computed based on the defined preamble signals. We analyzed the performance to clarify the theoretical limit. The feasibility of the proposed idea was confirmed with experimental results.

Gait rehabilitation commonly relies on bodyweight unloading mechanisms, such as overhead mechanical support and underwater buoyancy. Lightweight and wireless inertial measurement unit (IMU) sensors provide a cost-effective tool for quantifying body segment motions without the need for video recordings or ground reaction force measures. Identifying the instant when the foot contacts and leaves the ground from IMU data can be challenging, often requiring scrupulous parameter selection and researcher supervision. We aimed to assess the use of machine learning methods for gait event detection based on features from foot segment rotational velocity using foot-worn IMU sensors during bodyweight-supported treadmill walking on land and underwater. Twelve healthy subjects completed on-land treadmill walking with overhead mechanical bodyweight support, and three subjects completed underwater treadmill walking. We placed IMU sensors on the foot and recorded motion capture and ground reaction force data on land and recorded IMU sensor data from wireless foot pressure insoles underwater. To detect gait events based on IMU data features, we used random forest machine learning classification. We achieved high gait event detection accuracy (95–96%) during on-land bodyweight-supported treadmill walking across a range of gait speeds and bodyweight support levels. Due to biomechanical changes during underwater treadmill walking compared to on land, accurate underwater gait event detection required specific underwater training data. Using single-axis IMU data and machine learning classification, we were able to effectively identify gait events during bodyweight-supported treadmill walking on land and underwater. Robust and automated gait event detection methods can enable advances in gait rehabilitation.