Error Improvement in Visual Odometry Using Super-resolution

Multi-frame super-resolution recovers a high-resolution (HR) image from a sequence of low-resolution (LR) images. In this paper, we propose an algorithm that performs multi-frame super-resolution in an online fashion. This algorithm processes only one low-resolution image at a time instead of co-processing all LR images which is adopted by state-of-the-art super-resolution techniques. Our algorithm is very fast and memory efficient, and simple to implement. In addition, we employ a noise-adaptive parameter in the classical steepest gradient optimization method to avoid noise amplification and overfitting LR images. Experiments with simulated and real-image sequences yield promising results.

The YTU dataset and recurrent neural network based visual-inertial odometry

The current visual simultaneous localization and mapping (SLAM) systems require the use of matched feature point pairs to estimate camera pose and construct environmental maps. Therefore, they suffer from poor performance of the visual feature matchers. To address this problem, a visual SLAM using deep feature matcher is proposed, which is mainly composed of three parallel threads: Visual Odometry, Backend Optimizer and LoopClosing. In the Visual Odometry, the deep feature extractor with convolutional neural networks is utilized for extracting feature points in each image frame. Then, the deep feature matcher is used for obtaining the corresponding feature landmark pairs. Afterwards, a fusion method based on the last and the reference frame is proposed for camera pose estimation. The Backend Optimizer is designed to execute local bundle adjustment for a part of camera poses and landmarks (map points). While LoopClosing, consisting of a lightweight deep loop closure detector and the same matcher as the one used in Visual Odometry is utilized for loop correction based on pose graph. The proposed system has been tested extensively on most of benchmark KITTI odometry dataset. The experimental results show that our system yields better performance than the existing visual SLAM systems. It can not only run in real-time at a speed of 0.08 s per frame, but also reduce estimation error by at least 0.1 m.

Visual odometry is an essential component in robot navigation and autonomous driving; however, visual sensors are vulnerable in fast motion or sudden illumination changes. This weakness can be compensated with inertial measurement units (IMUs), which maintain the short-term motion when visual sensing becomes unstable and enhance the quality of estimated motion with inertial information. An omnidirectional multi-view visual odometry (ROVO) has recently demonstrated superior performance and stability with the unceasing feature observation of the omnidirectional setup; however, the shortcomings of visual odometry remain. This letter introduced an omnidirectional visual-inertial odometry system (ROVINS) that could seamlessly integrate the inertial information into the omnidirectional visual odometer algorithm: (a) The soft relative pose constraints from the inertial measurement are first added to the pose optimization formulation, which enables blind motion estimation when all visual features are lost; (b) Using the prediction results from the estimated velocity, the visual features in tracking are initialized, resulting in feature tracking that is more robust to visual disturbances. The experimental results showed that the proposed ROVINS algorithm outperforms the vision-only algorithm by a significant margin.

Generating a rendered image sequence through Monte Carlo ray tracing is an appealing option when one aims to accurately simulate various lighting effects. Unfortunately, interactive rendering scenarios limit the allowable sample size for such sampling-based light transport algorithms, resulting in an unbiased but noisy image sequence. Image denoising has been widely adopted as a post-sampling process to convert such noisy image sequences into biased but temporally stable ones. The state-of-the-art strategy for interactive image denoising involves devising a deep neural network and training this network via supervised learning, i.e., optimizing the network parameters using training datasets that include an extensive set of image pairs (noisy and ground truth images). This paper adopts the prevalent approach for interactive image denoising, which relies on a neural network. However, instead of supervised learning, we propose a different learning strategy that trains our network parameters on the fly, i.e., updating them online using runtime image sequences. To achieve our denoising objective with online learning, we tailor local regression to a cross-regression form that can guide robust training of our denoising neural network. We demonstrate that our denoising framework effectively reduces noise in input image sequences while robustly preserving both geometric and non-geometric edges, without requiring the manual effort involved in preparing an external dataset.

In the real world, there are a variety of applications requiring high resolution (HR) images in remote sensing, medicine, military information acquisition, multimedia, etc. Because of the high cost and physical limitations of the acquisition hardware, the low resolution (LR) images are used frequently. So, super resolution (SR) restoration is an emerged solution permitting to form one or a set of HR images from a sequence of LR images. In this paper, a novel SR restoration algorithm based on wavelet decomposition was proposed and finally has been implemented on OMAP3530 platform, demonstrating the effectiveness of the reconstruction of SR images. Quantitative and qualitative experiments for the introduced method showed great superiority in obtaining high resolution images with high PSNR and much better visual quality.

For their complete realization, autonomous vehicles (AVs) fundamentally rely on the Global Navigation Satellite System (GNSS) to provide positioning and navigation information. However, in area such as urban cores, parking lots, and under dense foliage, which are all commonly frequented by AVs, GNSS signals suffer from blockage, interference, and multipath. These effects cause high levels of errors and long durations of service discontinuity that mar the performance of current systems. The prevalence of vision and low-cost inertial sensors provides an attractive opportunity to further increase the positioning and navigation accuracy in such GNSS-challenged environments. This paper presents enhancements to existing multisensor integration systems utilizing the inertial navigation system (INS) to aid in Visual Odometry (VO) outlier feature rejection. A scheme called Aided Visual Odometry (AVO) is developed and integrated with a high performance mechanization architecture utilizing vehicle motion and orientation sensors. The resulting solution exhibits improved state covariance convergence and navigation accuracy, while reducing computational complexity. Experimental verification of the proposed solution is illustrated through three real road trajectories, over two different land vehicles, and using two low-cost inertial measurement units (IMUs).

The most significant part of any autonomous intelligent robot is the localization module that gives the robot knowledge about its position and orientation. This knowledge assists the robot to move to the location of its desired goal and complete its task. Visual Odometry (VO) measures the displacement of the robots’ camera in consecutive frames which results in the estimation of the robot position and orientation. Deep Learning, nowadays, helps to learn rich and informative features for the problem of VO to estimate frame-by-frame camera movement. Recent Deep Learning-based VO methods train an end-by-end network to solve VO as a regression problem directly without visualizing and sensing the label of training data in the training procedure. In this paper, a new approach to train Convolutional Neural Networks (CNNs) for the regression problems, such as VO, is proposed. The proposed method first changes the problem to a classification problem to learn different subspaces with similar observations. After solving the classification problem, the problem converts to the original regression problem to solve using the knowledge achieved by solving the classification problem. This approach helps CNN to solve regression problem globally in a local domain learned in the classification step, and improves the performance of the regression module for approximately 10%.

The problem of moving point target recognition based on noisy image sequence is discussed. A novel method for moving point target recognition is proposed. This method consists of a series of three major processes: preprocessing of the noisy image sequence; extracting the velocity of the moving point target; and neural classification. A simulation experiment was carried out for three kinds of moving point targets for which the signal-to-noise ratio (SNR) is specified. The experimental results indicate that the proposed method is feasible. The application of the neural network to classification greatly improves the classification results. >

The paper considers the problem of multi-frame super-resolution under applicative noise which generates distributed regions of outlying observations (damaged regions) in initial low resolution images. It analyses the existing solutions to the problem based on using recurrent algorithms of optimal filtering of a sequence of low resolution images. The paper also considers the existing super-resolution algorithms based on convolutional neural networks and deep learning models. A new approach and a corresponding image processing scheme is suggested for multi-frame image super-resolution under applicative noise based on using a convolutional neural network. The paper presents the results of the experiment conducted in order to compare the considered approaches to multi-frame image super-resolution by means of a sequence of low resolution frames under applicative noise.

In the field of simultaneous localization and mapping (SLAM), visual odometry (VO) always has great application prospects. In recent years, with the progress in the field of machine learning, methods based on neural networks are constantly being updated and applied. In this paper, we propose a continuous and generalized monocular visual odometry method based on features and neural networks. First, the feature information of adjacent image sequences is extracted by matching and troubleshooting algorithm (FLANN_PSC-RANSAC), then it and the corresponding six-degree-of-freedom information are simultaneously input into the long short-term memory artificial neural network (LSTM) for model construction, which not only ensures the reliability of the mode but also eliminates the influence of illumination on the data. In the real environment test, it has been effectively proved in terms of trajectory recovery accuracy and generalization ability to different environments and different illuminations.

Face recognition in low-resolution images has seen significant advancements over the past few decades. Although extensive research has been conducted to improve accuracy in these conditions, one of the main challenges remains the difficulty in identifying unique facial features in low-resolution images, leading to high error rates in identification. The use of Deep Convolutional Neural Networks (DCNN) for low-resolution face recognition is still limited. However, employing super-resolution models like REAL-ESRGAN can enhance recognition accuracy in low-resolution images. This study utilizes the Light CNN architecture and applies the margin-based identity loss function AdaFace on low-resolution datasets. The model is trained using the Casia-WebFace dataset and evaluated using the LFW and TinyFace test datasets. Based on the evaluation results on the LFW test data, the best model is Light CNN9-AdaFace, achieving the highest accuracy of 97.78% at 128x128 resolution. For images with the lowest resolution of 16x16, an accuracy of 83.37% was achieved using super-resolution techniques. On the TinyFace test data, the use of super-resolution resulted in performance metrics with a Rank-1 accuracy of 47.26%, Rank-5 accuracy of 55.25%, Rank-10 accuracy of 58.61%, and Rank-20 accuracy of 61.90% using the Light CNN9-AdaFace architecture.

The image degradation caused by motion blur, non-ideal sample and noise was producing in the process of Image acquirement. This paper proposed a fast super-resolution image reconstruction algorithm basing on image sequences. On the basis of image registration, registration algorithm used affine transform as geometric transform Model. A sequence of low-resolution images was roughly registered basing on feature and then use registration algorithm basing on Gray to optimize the result. Iterative back-projection technique was used to construct high resolution from image sequences. Firstly it made common low-resolution sequence images relate to standard displacements, and then reconstructed high-resolution image according to the relationship between low-resolution sequence images with standard displacement and high-resolution image. The high-frequency was distilled through the local estimation. By compensating the high-frequency component, the high-resolution images were recovered. Experimental results show that this algorithm solve the problem that the translation and rotation is small in traditional method. It has characterized over low computation complexity, fast convergence. The details, definition and resolution of high resolution image processed with the proposed method are effectively improved.

Zoom based image super-resolution using DCT with LBP as characteristic model

In colonoscopy procedures, it is important to rebuild and visualize the colonic surface to minimize the missing regions and reinspect for abnormalities. Due to the fast camera motion and deformation of the colon in standard forward-viewing colonoscopies, traditional simultaneous localization and mapping (SLAM) systems work poorly for 3D reconstruction of colon surfaces and are prone to severe drift. Thus in this paper, a preoperative colon model segmented from CT scans is used together with the colonoscopic images to achieve the 3D colon reconstruction. The proposed framework includes dense depth estimation from monocular colonoscopic images using a deep neural network (DNN), visual odometry (VO) based camera motion estimation and an embedded deformation (ED) graph based non-rigid registration algorithm for deforming 3D scans to the segmented colon model. A realistic simulator is used to generate different simulation datasets with ground truth. Simulation results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. In-vivo experiments are also conducted and the results show the practicality of the proposed framework for providing useful shape and texture information in colonoscopy applications.