Visual odometry errors and fault distinction for integrity monitoring

Visual odometry is critical in visual simultaneous localization and mapping for robot navigation. However, the pose estimation performance of most current visual odometry algorithms degrades in scenes with unevenly distributed features because dense features occupy excessive weight. Herein, a new human visual attention mechanism for point-and-line stereo visual odometry, which is called point-line-weight-mechanism visual odometry (PLWM-VO), is proposed to describe scene features in a global and balanced manner. A weight-adaptive model based on region partition and region growth is generated for the human visual attention mechanism, where sufficient attention is assigned to position-distinctive objects (sparse features in the environment). Furthermore, the sum of absolute differences algorithm is used to improve the accuracy of initialization for line features. Compared with the state-of-the-art method (ORB-VO), PLWM-VO show a 36.79% reduction in the absolute trajectory error on the Kitti and Euroc datasets. Although the time consumption of PLWM-VO is higher than that of ORB-VO, online test results indicate that PLWM-VO satisfies the real-time demand. The proposed algorithm not only significantly promotes the environmental adaptability of visual odometry, but also quantitatively demonstrates the superiority of the human visual attention mechanism.

Visual odometry systems are usually categorized as feature-based visual odometry systems and direct visual odometry systems. The feature-based visual odometry systems spend time on calculating descriptors while direct visual odometry systems directly calculate residuals. The optimization of direct visual odometry systems is sensitive with initial states while feature-based visual odometry systems did not suffer from this problem. This paper presents a features assisted stereo direct visual odometry system with virtual wide field-of-view tracking. It combines the advantages of feature-based methods and direct methods. It is achieved by incorporating a feature assisted predicting module and a virtual wide field-of-view tracking module into stereo direct sparse odometry (SDSO) framework. The feature-based matching is combined in direct tracking module for efficient and robust data association. Furthermore, before data association, a virtual wide field-of-view inverse depth frame is created in case of tracking lost in aggressive rotating conditions. Evaluation results on the EuRoC datasets and the KITTI dataset show that, with the aid of the new tracking module, the stereo visual odometry becomes more robust and more accurate.

Unsupervised monocular visual odometry with decoupled camera pose estimation

Stereo-vision devices have rigorous requirements for extrinsic parameter calibration. In Stereo Visual Inertial Odometry (VIO), inaccuracy in or changes to camera extrinsic parameters may lead to serious degradation in estimation performance. In this manuscript, we propose an online calibration method for stereo VIO extrinsic parameters correction. In particular, we focus on Multi-State Constraint Kalman Filter (MSCKF [1]) framework to implement our method. The key component is to formulate stereo extrinsic parameters as part of the state variables and model the Jacobian of feature reprojection error with respect to stereo extrinsic parameters as sub-block of update Jacobian. Therefore we can estimate stereo extrinsic parameters simultaneously with inertial measurement unit (IMU) states and camera poses. Experiments on EuRoC dataset and real-world outdoor dataset demonstrate that the proposed algorithm produce higher positioning accuracy than the original S-MSCKF [2], and the noise of camera extrinsic parameters are self-corrected within the system.

IntroductionSimulating thermal convection in Earth and planetary interiors is crucial for various applications, including benchmarking numerical codes (e.g. [1], [2]), deriving scaling laws for heat transfer in complex flows (e.g. [3], [4]), determining mixing efficiency (e.g. [5], [6]) and the characteristic spatial wavelengths of convection (e.g. [7]). However, reaching a statistically-steady state in these simulations, even in 2D, can be computationally expensive. While choosing "close-enough" initial conditions can significantly speed simulations up, this selection process can be challenging, especially for systems with multiple controlling parameters. This work explores how machine learning can be leveraged to identify optimal initial conditions, ultimately accelerating numerical convection simulations on their path to a statistically-steady state.Convection model and neural network architectureWe have compiled a comprehensive dataset comprising 128 simulations of Rayleigh-Bénard convection within a rectangular domain of aspect ratio 4, characterized by variable viscosity and internal heating. A randomized parameter sweep was employed to vary three key factors: the ratio of internal heat generation to thermal driving forces (RaQ/Ra), the temperature-dependent viscosity contrast (FKT), and the pressure-dependent viscosity contrast (FKV). Each simulation was executed until the root mean square velocity and mean temperature attained a statistically stationary state. Subsequently, we extracted the time-averaged one-dimensional temperature profile, horizontally averaged across the domain, over the final 200 time steps of each simulation.Figure 1: Schematic of the neural network used to predict statistically-steady temperature profiles.We used a feedforward neural network (NN) to predict the 1D temperature profiles as a function of the control parameters (Fig. 1). We modified the formulation of the NN from the one we used in [8] in the following ways. First, instead of each training example consisting of the full temperature profile, we now predict temperature at a given spatial point in depth (y). This allows the network to learn and predict profiles on variable computational grids, making it more adaptable. This formulation also avoids the "wiggles" encountered when predicting high-dimensional profiles. Second, we use skip-connections. For a given hidden layer, the features from all the previous hidden layers are added to it before applying the activation function. Third, we use SELU instead of tanh for faster convergence. Fourth, we concatenate the parameters to the last hidden layer, a technique shown to improve learning in some cases  (e.g., [9]). Fifth, as a consequence of conditioning the NN on spatial points, we augment the data with copies of points from the bottom and top thermal boundary layers to ensure that each batch is likely to be more representative of relevant temperature features occurring at the outer boundaries of the domain. We split the dataset into training, cross-validation, and test sets with 96, 15, and 16 simulations, respectively. To test the extrapolation capacity of the network, we pick simulations for the test set only where at least one of the simulation parameters exceeds a certain threshold. The cross-validation dataset consists of interpolated values only and is used additionally to choose the best performing NN architecture and save network weights only when the cross-validation loss improves. ResultsWe test the NN against the following baselines: linear regression, kernel ridge regression, and nearest neighbor. As shown in Table 1, when predicting unseen profiles from interpolated cases in the cross-validation set or even the extrapolated cases (Test set), the NN predictions are the most accurate among the tested possibilities. We visualize some of the profiles predicted by these algorithms in the cross-validation set in Fig. 2. Linear regression results are left out for ease of visualization.  Algorithm                  Training set Cross-validation set Test set Linear Regression  0.0385 0.0388 0.0676 Kernel Ridge Regression  0.0148 0.0147 0.0371 Neural Network 0.0071  0.0071  0.0187 Nearest Neighbor   0.0 0.0282  0.0495 Table 1: Mean absolute error of the prediction of 1D temperature profiles of statistically-steady simulations based on different tested algorithms. The lowest error per dataset is highlighted in bold.The plots in Fig. 2 corroborate the obtained statistics about the accuracy of different methods. When the simulation parameters we are predicting happen to be close to a simulation in the training set, the nearest neighbor profile already provides a good estimate. Otherwise, the NN predictions appear to be the most accurate. This is encouraging because it shows that even for small datasets such as the one we considered, machine learning can already start delivering useful results. Figure 2: Four temperature profiles from the cross-validation dataset (solid red lines) and the corresponding predictions from different models (dashed lines).OutlookWe are currently exploring how initial conditions impact the time a system takes to reach a statistically-steady state in terms of temperature and RMS velocity. We're comparing four starting points: hot, cold, linear temperature profiles, and predictions from our neural network. This will quantify the efficiency gain achieved by using the NN predictions to accelerate simulations towards equilibrium. In addition, we also plan to share the dataset and trained neural network with the community.

The correspondence of a sensory object to the category of a descriptive statement requires a reflexive-identity of the object, and such a reflexive-identity is primarily based on the cognition of spatiality. Spatiality is, however, constituted through visual perception. There are only two occasions on which definitive reflexive-identity is exemplified: the infinitesimal point and the infinite “One,” and others are just human stipulations that meet pragmatic needs of rough identification of things at hand. However, if a spatial point is not different from any so-called “other” spatial point, to validate the reflexive identity of any spatial point implies a validation of the reflexive identity of all spatial points. Thus, as “one” and “many” here become absolutely unitary, the infinitely small and the infinitely immense are identical.

We aim to estimate brain tissue displacements in the medial temporal lobe (MTL) using backscattered ultrasound radiofrequency (US RF) signals, and to assess the diagnostic ability of brain tissue displacement parameters for the differentiation of patients with Alzheimer’s disease (AD) from healthy controls (HC). Standard neuropsychological evaluation and transcranial sonography (TCS) for endogenous brain tissue motion data collection are performed for 20 patients with AD and for 20 age- and sex-matched HC in a prospective manner. Essential modifications of our previous method in US waveform parametrization, raising the confidence of micrometer-range displacement signals in the presence of noise, are done. Four logistic regression models are constructed, and receiver operating characteristic (ROC) curve analyses are applied. All models have cut-offs from 61.0 to 68.5% and separate AD patients from HC with a sensitivity of 89.5% and a specificity of 100%. The area under a ROC curve of predicted probability in all models is excellent (from 95.2 to 95.7%). According to our models, AD patients can be differentiated from HC by a sharper morphology of some individual MTL spatial point displacements (i.e., by spreading the spectrum of displacements to the high-end frequencies with higher variability across spatial points within a region), by lower displacement amplitude differences between adjacent spatial points (i.e., lower strain), and by a higher interaction of these attributes.

With the development of fingerprinting-based visual localization technology, a problem with this method is standing out, which is it takes great expenses in fingerprint collection. Recently, a few studies proposed some methods to alleviate this problem. However, the accuracy of the existing method is relatively low under some scenarios such as wide field of vision. In this paper, we propose a novel automatic visual fingerprinting (AVF) method for an indoor visual localization system. We consider the performance of AVF greatly hinges on visual odometry (VO) and ego-motion estimation (EME) block, which are two different ways of estimating fingerprint coordinates. Since both visual odometry and ego-motion estimation model are inaccurate, we build the least square model by second-order cone programming (SOCP). Our SOCP-based method is proposed to deal with the serious cumulative error and the random error introduced by VO and EME model, respectively. The purpose of this paper is improving the accuracy of the database generated by the AVF method under wide field of vision scenarios. Although the time costs are relatively higher than our compared method, fortunately, it is only on the off-line stage. The simulation results show that our method can provide a reliable image-location database with the consumer-grade smartphone camera.

The localization system is the most important part of the overall drone navigation system. The Global Positioning System (GPS) or Global Navigation Satellite System (GNSS) is the main device commonly used in a drone. However, under certain conditions, GPS or GNSS may not function optimally, such as in situations of signal jamming or enclosed environments. This paper implemented a new approach to address this issue by combining GNSS data with Visual Odometry (VO) through Machine Learning (ML) methods. The followed process consists of three main stages. First, performing speed and orientation estimation using VO. Second, performing left and right feature separation on the images to generate a more stable and robust estimation of speed and rotation. Third, refining speed and orientation estimation by integrating GNSS data through ML-based data fusion. The proposed method strives to enhance drone localization accuracy, despite disruptions or unavailability of GNSS signals. The research results indicate that the introduced method significantly reduces Absolute Translation Error (ATE) compared to utilizing VO or GNSS separately. The average ATE produced reached 4.38 m and an orientation of 8.26°, indicating that this data fusion approach provides a significant improvement in drone localization accuracy, making it reliable in operational scenarios with limited GNSS signals.

In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

As the front end of visual simultaneous localization and mapping (vSLAM), the feature points matching of the visual odometry (VO) provides important robot pose judgment, while there are mismatches in the feature points matching due to inaccurate information obtained from cameras. In the process of feature points matching, VO initializes the matching of feature points through two adjacent frames of pictures, as SLAM runs, the cumulative error will gradually increase, resulting in mapping and positioning results inaccurate. Aiming to reduce the feature points matching error during initialization, an evidential fusion method for feature points matching is put forward. On the basis of descriptor similarity in Oriented FAST and Rotated BRIEF SLAM (ORB-SLAM) and pixel similarity around feature points, we treat the similarity index as an evidence, and take Belief Jensen-Shannon (BJS) divergence to measure distance between different evidences. The proposed evidential method considers the uncertainties brought by different indexes in accord with the feature points matching, and obtains suitable matching points reducing false matches through the combination of evidences. BJS divergence measure is carried out to measure the discrepancy and conflict degree among the evidences. After that, the reliability of evidence could be obtained to modify the information volume of the evidences. Finally, an example of feature points matching shows that the evidential method is feasible for feature points matching.

Recently, unsupervised methods for monocular visual odometry (VO), with no need for quantities of expensive labeled ground truth, have attracted much attention. However, these methods are inadequate for long-term odometry task, due to the inherent limitation of only using monocular visual data and the inability to handle the error accumulation problem. By utilizing supplemental low-cost inertial measurements, and exploiting the multi-view geometric constraint and sequential constraint, an unsupervised visual-inertial odometry framework (UnVIO) is proposed in this paper. Our method is able to predict the per-frame depth map, as well as extracting and self-adaptively fusing visual-inertial motion features from image-IMU stream to achieve long-term odometry task. A novel sliding window optimization strategy, which consists of an intra-window and an inter-window optimization, is introduced for overcoming the error accumulation and scale ambiguity problem. The intra-window optimization restrains the geometric inferences within the window through checking the photometric consistency. And the inter-window optimization checks the 3D geometric consistency and trajectory consistency among predictions of separate windows. Extensive experiments have been conducted on KITTI and Malaga datasets to demonstrate the superiority of UnVIO over other state-of-the-art VO / VIO methods. The codes are open-source.

When disaster strikes in urban areas, the devastating results are collapsed structures that may contain voids, and trapped people within. To a large extent, the speed with which these victims can be found and extricated determines the likelihood of their survival. Specially trained and equipped emergency first responders are tasked with trying to save their lives by locating and extricating trapped victims from these dangerous environments. Telepresence systems can help first responders search for casualties from a safe location. Most automated search systems intended for use in urban disasters, come in the form of remotely operated robots. This work takes a different approach to telepresence and robotics. This work is an extension of previous work that exploits the intelligence and characteristics of trained search dogs combined with compatible technology and used as components in new kinds of telepresence systems for urban search and rescue (USAR) operations. The Canine Remote Deployment System (CRDS) is a tool that emergency responders can use to deliver critical supplies to trapped victims in rubble using dogs. The first contribution of this work is the development of the bark detection system for automatically triggering deployment of packages near trapped victims from the CRDS-guaranteeing accurate package deployment even when remote communication with the dog is impossible. A well-known ground robot problem is the difficulty in designing a mobility mechanism to traverse rubble. Another contribution of this thesis is the Canine Assisted Robot Deployment (CARD) framework and the design of a robot capable of being carried by a search dog. This work extends the responder’s telepresence in rescue operations by bringing robots much deeper into the disaster site than current methods. Visual odometry is used in location tracking in GPS-denied environments and can be used in rescue operations. This research explores the limitation of RGB-D cameras for visual odometry for this application. An algorithm called pseudo-Random Interest Points Extractor was developed iv to track images over visually feature-sparse areas with the potential use of visually reconstructing canine search paths to victims. This work concentrates on using visual odometry from data collected from a search dog-mounted RGB-D camera. The task of model stabilization is difficult due to the nature of dog’s constant and unpredictable movements, asthe data contains many motion blurred images. The development of an algorithm called Intelligent Frame Selector is shown to improve visual odometry for systems carried by search dogs by intelligently filtering data and selecting only usable frames. The algorithm can be applied to any general visual odometry pipeline beneficially as the technique reduces cumulative error problems by using less data.

With the rapid increase of urban electricity demand in recent years, the scale and number of urban power plants are expanding, which poses new challenges to the safety and management of power plant staff, and real-time high-precision localization services can achieve fine-grained management of staff. We propose a vision fusion-based localization technique that combines the traditional visual odometry calculation method based on the feature point method with the optical flow method, and validates it by collecting data sets in a real power plant environment. The experimental results show that the proposed fusion-based visual odometry calculation method is superior to the traditional visual odometry calculation method in terms of comprehensive performance in terms of real-time performance and accuracy of position estimation, which verifies the practical application possibilities of the vision-based localization technology in urban power plants.

Monocular Visual Odometry (VO) is widely used for small aerial platforms navigation. However, environment conditions can limit the use of cameras. Radar Odometry (RO) can be envisaged when illumination conditions hinder visual navigation. Indeed, radar can help estimation of platform speed and interframe platform displacement, thus solving also the inherent monocular VO scale estimation problem. Performance of radar-aided monocular VO navigation solution, to be used then within the framework of all-illumination multi-sensor navigation, is assessed with test in real operational scenario. Results show acceptable standalone performance, that can be further improved with more complex architectures.