Towards Privacy-Preserving Ego-Motion Estimation Using an Extremely Low-Resolution Camera

The application of unmanned vehicles in civilian and military fields is increasingly widespread. Traditionally, unmanned vehicles primarily rely on Global Positioning Systems (GPSs) for positioning; however, GPS signals can be limited or completely lost in conditions such as building obstructions, indoor environments, or electronic interference. In addition, countries are actively developing GPS jamming and deception technologies for military applications, making precise positioning and navigation of unmanned vehicles in GPS-denied or constrained environments a critical issue that needs to be addressed. In this work, authors propose a method based on Visual–Inertial Odometry (VIO), integrating the extended Kalman filter (EKF), an Inertial Measurement Unit (IMU), optical flow, and feature matching to achieve drone localization in GPS-denied environments. The proposed method uses the heading angle and acceleration data obtained from the IMU as the state prediction for the EKF, and estimates relative displacement using optical flow. It further corrects the optical flow calculation errors through IMU rotation compensation, enhancing the robustness of visual odometry. Additionally, when re-selecting feature points for optical flow, it combines a KAZE feature matching technique for global position correction, reducing drift errors caused by long-duration flight. The authors also employ an adaptive noise adjustment strategy that dynamically adjusts the internal state and measurement noise matrices of the EKF based on the rate of change in heading angle and feature matching reliability, allowing the drone to maintain stable positioning in various flight conditions. According to the simulation results, the proposed method is able to effectively estimate the flight trajectory of drones without GPS. Compared to results that rely solely on optical flow or feature matching, it significantly reduces cumulative errors. This makes it suitable for urban environments, forest areas, and military applications where GPS signals are limited, providing a reliable solution for autonomous navigation and positioning of drones.

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

Nowadays, multi-sensor fusion technology is a fundamental prerequisite to achieve highly autonomous robots and robustness. Many studies have been conducted, such as visual–inertial odometry (VIO), integrated navigation, and LiDAR–inertial odometry. Typically, for VIO, gratifying results have been achieved, ascribing to the complementary sensing capabilities of inertial measurement units (IMUs) and cameras. However, this work mainly focuses on the fusion of visual and inertial data, while the IMU error is less considered, especially for low-cost or poorly calibrated microelectromechanical system (MEMS) IMU. Such errors may have a significant effect on the VIO performance. In this study, we compensated for the IMU using camera assistance. The key characteristic of the method is that we optimize the compensation parameters (scale factor) from coarse to fine by combining the time domain with the frequency domain. The proposed method is to use the time-domain and frequency-domain optimization to suppress large noise in the dynamic calibration process of the extremely low-cost sensor platform. The effectiveness of this method is validated through experiments and simulations. The minimal calibration error (0.46%) is commensurate with the advanced work. By feeding the compensated IMU into the VIO algorithm, the localization accuracy is improved by 9% to 15%. This method improves the performance in the VIO algorithm, which is equipped with the low-cost or poorly calibrated MEMS IMU and reduces the hardware and deployment costs of the system.

Ground truth generation for optical flow is hard and costly. Real images can be annotated with ground truth optical flow, but two problems persist: first, no flow measurement techniques for dynamic, large-scale data exist. Second, real-world parameters such as weather or surface wetness cannot be systematically varied. On the other hand, rendered images can have perfect ground truth while light, geometry and materials can be systematically varied. But do they resemble reality well enough to serve as a basis for performance evaluation? In this paper, we take computer graphic realism to an extreme: we compare three optical flow results on reflectance field renderings with supposedly identical real images. A systematic variation of reflectance realism reveals that computer graphics are a valid way to create complex synthetic sequences for optical flow evaluation — but only if modeled carefully.

This paper reports an optical based method for quantification of cardiac motion in MRI images. The cardiac motion is quantified in a Short-Axis (SAX) slice located at the mid-cavity of the left ventricle. In this slice, the left ventricle wall is segmented for extracting the endocardium, the epicardium and the mid‑wall. The velocity field for these contours is available after performing the optical flow estimation for the given slice. Post-processing of this motion field enables estimation of the radial displacement. The core of this cardiac motion quantification method is a sparse based algorithm for optical flow estimation. This algorithm is presented and compared with several classical optical flow estimation algorithms. The comparison is performed using several video sequences available from the Middlebury benchmark where the ground truth optical flow is known. The algorithms with better performance are also tested using a 4‑ D cardiac Magnetic Resonance Image (MRI) sequence. Results show that he sparse algorithm estimates an optical flow field that represents the motion of contraction during the systole interval. Moreover, results about radial displacement estimation on real MRI sequences for a normal subject and a patient with hypertrophy show that the proposed quantification method could be useful for quantification of left ventricle motion.

Accurate motion estimation plays a crucial role in state estimation of an unmanned aerial vehicle (UAV). This is usually carried out by fusing the kinematics of an inertial measurement unit (IMU) with the video output of a camera. However, the accuracy of existing approaches is hindered by the discretization effect of the model even at a high IMU sampling rate. In order to improve the accuracy, we propose a new IMU motion integration model for the IMU kinematics in continuous time. The kinematics are modeled using a switched linear system. A closed-form discrete formulation is derived to compute the mean measurement, the covariance matrix, and the Jacobian matrix. Thus, it is more accurate and more efficient for online estimation of visual-inertial odometry (VIO), particularly when there is a high dynamic change in the agent's motion or the agent travels with high speed. The proposed IMU factor framework is evaluated using both real public datasets and indoor environment under different scenarios of motion capture. Our evaluation shows that the proposed framework outperforms the state-of-the-art VIO approach by up to 22.71% accuracy improvement on the EuRoc dataset and 38.15% accuracy improvement for motion estimation under the indoor environment.

In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

Cameras and inertial measurement units are complementary sensors for ego-motion estimation and environment mapping. Their combination makes visual-inertial odometry (VIO) systems more accurate and robust. For globally consistent mapping, however, combining visual and inertial information is not straightforward. To estimate the motion and geometry with a set of images large baselines are required. Because of that, most systems operate on keyframes that have large time intervals between each other. Inertial data on the other hand quickly degrades with the duration of the intervals and after several seconds of integration, it typically contains only little useful information. In this letter, we propose to extract relevant information for visual-inertial mapping from visual-inertial odometry using non-linear factor recovery. We reconstruct a set of non-linear factors that make an optimal approximation of the information on the trajectory accumulated by VIO. To obtain a globally consistent map we combine these factors with loop-closing constraints using bundle adjustment. The VIO factors make the roll and pitch angles of the global map observable, and improve the robustness and the accuracy of the mapping. In experiments on a public benchmark, we demonstrate superior performance of our method over the state-of-the-art approaches.

This dissertation presents a comprehensive approach to addressing the challenges associated with the control and navigation of Unmanned Aerial Vehicles (UAVs), specifically focusing on robustness under GPS-denied, turbulent, and dark indoor conditions with various disturbances. Initial efforts involved designing and theoretically validating Sliding Mode Controllers (SMC) with Disturbance Observers using MATLAB/SIMULINK, optimized using offline Particle Swarm Optimization (PSO). In order to evaluate their performance, a lightweight simulator was developed to simulate both quadcopter and hexacopter configurations under the PX4 architecture. This simulator served to validate and compare the optimized SMC against PSO-optimized PID controllers. The results of these comparisons demonstrated the superior adaptability of the SMC, especially in environments characterized by significant sensor noise and disturbances. Subsequent practical indoor tests were conducted with real quadcopters, which provided additional verification of the robustness and reliability of the enhanced control theory with the actual UAVs under dynamic disturbances. To further evaluate the compatibility and adaptability of the SMC, this dissertation establishes a framework for interfacing with high-performance STMH7 chips, serving as the UAV microcontroller unit (MCU). This framework facilitates the connection of various sensors and provides guidance for constructing different quadcopter frames. Moreover, it enables the comparison of different flight controllers on distinct air-frames. Concurrently, the optimized SMC allows for remote dynamic parameter tuning during flight. In pursuit of improved localization capacity, the dissertation also explores the integration of an Intel Realsense D435i camera with an Inertial Measurement Unit (IMU) for Visual-Inertial Odometry (VIO) fusion within the Robot Operating System (ROS) environment. This integration employs an onboard companion computer for real-time VIO estimation and flight command. The effectiveness of the VIO algorithm is verified through both simulations and real-world experiments, highlighting its ability to enhance UAV localization and navigation. Additionally, a visual inertial odometry method for UAV Simultaneous Localization and Mapping (vSLAM) is tested and validated in both simulated and real drone environments. Furthermore, the dissertation introduces an innovative hybrid filtered multi-directional radar inertial odometry solution denoted as Hybrid-MRIO. This Error State Extended Kalman Filter (ES-EKF) based Radar Inertial Odometry (RIO) approach is implemented within the open-source PX4 autopilot system, enhancing UAV navigation accuracy and performance, and serves as a robust navigation alternative, particularly in challenging environments where traditional navigation systems prove ineffective, such as dark or smoky conditions. This solution integrates data from multiple software synchronized high-resolution Frequency-Modulated Continuous-Wave (FMCW) mmWave radars with an IMU. A subsequent 4D (x, y, z, doppler) mmWave radar SLAM (rSLAM) system has been developed, including three modules: front-end, loop detection module, and back-end. In the front-end, radar ego-velocity is utilized for states estimation and dynamic object removal, and a point cloud registration-based approach known as APDGICP (Adaptive Probability Distribution-GICP) is employed for keyframe detection. The loop detection module utilizes the intensity scan context to identify potential loop closure candidates. In the back-end, a pose graph is constructed, integrating RIO estimated odometry and identified loop closures for better localization. To develop fully-autonomous UAVs with their decision making modules, this dissertation details the development of a custom AI-powered fully-autonomous quadcopter equipped with companion computers and stereo IR cameras. Additionally, this dissertation discusses a swarm of small Unmanned Aerial Systems (SUAS) interconnected through a software-defined communications network. These drones demonstrated their capacity to independently/collaboratively execute a wide range of tasks, including target search, detection, identification, classification, tracking, and following, both in simulated and real-world scenarios. The system exhibits advanced collision avoidance capabilities and the ability to strategically respond to dynamic scenarios, such as changes in target behavior, highlighting its effectiveness in managing dynamic and challenging situations without the need for continuous human intervention.--Author's abstract

The biomechanical mechanisms of swimming involve a number of aspects. The forces exerted by muscles during different swimming postures are crucial. These muscle contractions and relaxations follow specific biomechanical principles. This work aims to develop a swimming posture recognition system based on inertial sensors and a Convolutional Neural Network-Support Vector Machine (CNN-SVM) to improve the accuracy and real-time performance of posture recognition. First, an inertial sensor system to be worn on swimwear is designed to collect three-axis motion data, including acceleration, angular velocity, and magnetometer readings. The collected data are then preprocessed through denoising, normalization, and feature extraction steps to ensure high-quality input data. Next, a Convolutional Neural Network (CNN) is constructed to automatically extract high-level features from the preprocessed sensor data. The CNN model, through multi-layer convolution and pooling operations, effectively captures the spatiotemporal patterns in the motion data, extracting highly distinguishable features for posture recognition. To further improve the model’s classification performance, a Support Vector Machine (SVM) classifier is applied based on the CNN model. Specifically, CNN is responsible for feature extraction, while the SVM handles the final posture classification. Cross-validation is used to train and validate the model, assessing its performance. Experimental results show that the model achieves a 95% accuracy rate on the training dataset and maintains an accuracy rate above 93% on the test dataset. The system can accurately and in real-time recognize various swimming postures, including freestyle, breaststroke, backstroke, and butterfly. The recognition accuracy for all four swimming styles exceeds 91%. Understanding these biomechanical mechanisms helps in improving the accuracy of the recognition system. In summary, the proposed method for swimming posture recognition based on inertial sensors and CNN-SVM has significant advantages in accuracy and real-time performance. It allows for better interpretation of the sensor data and more precise identification of different postures. The high accuracy and generalization ability of the proposed system suggest that it can effectively capture and analyze the biomechanical nuances of swimming, providing valuable insights for swimming training and performance evaluation, and opening up new avenues for intelligent sports monitoring. evaluation.

Stereo-vision devices have rigorous requirements for extrinsic parameter calibration. In Stereo Visual Inertial Odometry (VIO), inaccuracy in or changes to camera extrinsic parameters may lead to serious degradation in estimation performance. In this manuscript, we propose an online calibration method for stereo VIO extrinsic parameters correction. In particular, we focus on Multi-State Constraint Kalman Filter (MSCKF [1]) framework to implement our method. The key component is to formulate stereo extrinsic parameters as part of the state variables and model the Jacobian of feature reprojection error with respect to stereo extrinsic parameters as sub-block of update Jacobian. Therefore we can estimate stereo extrinsic parameters simultaneously with inertial measurement unit (IMU) states and camera poses. Experiments on EuRoC dataset and real-world outdoor dataset demonstrate that the proposed algorithm produce higher positioning accuracy than the original S-MSCKF [2], and the noise of camera extrinsic parameters are self-corrected within the system.

The YTU dataset and recurrent neural network based visual-inertial odometry

The point feature, whose dynamic robustness has been widely studied, dominates in the field of visual simultaneous localization and mapping (SLAM) or visual-inertial odometry (VIO). When discussing a dynamic scene, line features are not given enough attention. This paper proposes a pre-processing step for VIO to reduce the influence of lines upon dynamic objects on system robustness and merges it into a state-of-the-art optimization-based VIO pipeline. First, it is determined whether the line feature is a potential dynamic line based upon the result of the semantic segmentation, optical flow and re-projection error. Then, instead of filtering them out, the information matrixes of these line features in the optimization function is adjusted by a weight-based method according to their tracked size. A simulated challenged visual-inertial dataset is used to evaluate the proposed algorithm against other state-of-the-art methods. The results shows that proposed method can increase robustness to dynamic scenes and make the line-based map neater and more intuitive through avoiding the drawing of dynamic line features during the mapping procedure.

Visual–inertial odometry is an effective system for mobile robot navigation. This article presents an egomotion estimation method for a dual-sensor system consisting of a camera and an inertial measurement unit (IMU) based on the cubature information filter and H∞ filter. The intensity of the image was used as the measurement directly. The measurements from the two sensors were fused with a hybrid information filter in a tightly coupled way. The hybrid filter used the third-degree spherical-radial cubature rule in the time-update phase and the fifth-degree spherical simplex-radial cubature rule in the measurement-update phase for numerical stability. The robust H∞ filter was combined into the measurement-update phase of the cubature information filter framework for robustness toward non-Gaussian noises in the intensity measurements. The algorithm was evaluated on a common public dataset and compared to other visual navigation systems in terms of absolute and relative accuracy.

VIO-DualProNet: Visual-inertial odometry with learning based process noise covariance