Feature Matching and Tracking for Visual Odometry of Mobile Robot

For many image processing applications (such as feature tracking, object recognition, stereo matching and remote sensing), the technique known as Template Matching (TM) plays an important role for the localization and recognition of objects or patterns within a digital image. A TM approach seeks to find a position within a source image which yields to the best possible resemblance between a given sub-image (typically referred as image template) and a corresponding region of such source image. TM involves two critical aspects: similarity measurement and search strategy. In this sense, the simplest available TM method involves an exhaustive computation of the Normalized Cross-Correlation (NCC) value (similarity measurement) over all pixel locations of the source image (search strategy). Unfortunately, this approach is strongly restricted due to the high computational cost implied in the evaluation of the NCC coefficient. Recently, several TM methods based on evolutionary approaches have been proposed as an alternative to reduce the number of search locations in the TM process. However, the lack of balance between exploration and exploitation related to the operators employed by many of such approaches makes TM to suffer from several critical flaws, such as premature convergence. In the proposed approach, the swarm optimization method known as Locust Search (LS) is applied to solve the problem of template matching. The unique evolutionary operators employed by LS method’s search strategy allows to explicitly avoid the concentration of search agents toward the best-known solutions, which in turn allows a better exploration of the valid image’s search region. Experimental results show that, in comparison to other similar methods, the proposed approach achieves the best balance between estimation accuracy and computational cost.

Auto Die Bonder (ADB) is often encountered in the IC package field, whose key technical difficulty lies in quickly and accurately locating the bonding positions of the chips. In this paper, a normalized cross correlation (NCC) algorithm based on image pixels has been used to address this problem. By using iterated algorithm with simplified expression, the modified NCC (MNCC) is remarkably accelerated compared to the traditional one. Afterwards, the MCNN algorithm has been examined by experiments to locate the lead-frame pads, and the result shows its superiority in efficiency with satisfactory positioning accuracy.

Fast and robust feature tracking for 3D reconstruction

Recently, the event camera has become a popular and promising vision sensor in the research of simultaneous localization and mapping and computer vision owing to its advantages: low latency, high dynamic range, and high temporal resolution. As a basic part of the feature-based SLAM system, the feature tracking method using event cameras is still an open question. In this article, we present a novel asynchronous event feature generation and tracking algorithm operating directly on event-streams to fully utilize the natural asynchronism of event cameras. The proposed algorithm consists of an event-corner detection unit, a descriptor construction unit, and an event feature tracking unit. The event-corner detection unit addresses a fast and asynchronous corner detector to extract event-corners from event-streams. For the descriptor construction unit, we propose a novel asynchronous gradient descriptor inspired by the scale-invariant feature transform descriptor, which helps to achieve quantitative measurement of similarity between event feature pairs. The construction of the gradient descriptor can be decomposed into three stages: speed-invariant time surface maintenance and extraction, principal orientation calculation, and descriptor generation. The event feature tracking unit combines the constructed gradient descriptor and an event feature matching method to achieve asynchronous feature tracking. We implement the proposed algorithm in C++ and evaluate it on a public event dataset. The experimental results show that our proposed method achieves improvement in terms of tracking accuracy and real-time performance when compared with the state-of-the-art asynchronous event-corner tracker and with no compromise on the feature tracking lifetime.

Automatic orientation and 3D modelling from markerless rock art imagery

Extraction of micro-vibration information from a single-shot speckle image is realized base on the row-by-row exposure mechanism of a rolling shutter camera and the high intensity of laser speckle. Normalized cross correlation and cubic spline interpolation algorithm is firstly used between nonoverlapping gray scales of adjacent rows to infer micro-displacement information. With number of rows set as 342, wide frequency range from 300 Hz to 20 kHz sinusoid signals are recovered with relative errors smaller than 2%. For low frequency signals from 40 Hz to 300 Hz, the relative errors of the extracted frequency are smaller than 5%. The effects of number of sampling cycles (NSC) and sampling points in unit period on the system performance are investigated and high accuracy is obtained with NSC of 4 and sampling points of 3 per signal period. In a single shot image, spatial multi points vibration detection scheme is presented and experimentally realized. The representative signals of 5 kHz, 10 kHz and 20 kHz are successfully extracted in the multidimensional scenes of 1D, 2D and 3D respectively, and the relative errors are all within 0.5%. This scheme can be potentially used in air surveillance for wide range remote sensing of interested object or sounding.

In this paper we propose a fast normalized cross correlation (NCC) algorithm for pattern matching based on combining adaptive multilevel partition with the winner update scheme. This winner update scheme is applied in conjunction with an upper bound for the cross correlation derived from Cauchy-Schwarz inequality. To apply the winner update scheme, we partition the summation of cross correlation into different levels with the partition order determined by the gradient energies of the partitioned regions in the template. Thus, this winner update scheme can be employed to skip the unnecessary calculation. Experimental results show the proposed algorithm is very efficient for image matching under different lighting conditions.

Initializing and accelerating Stereo-DIC computation using semi-global matching with geometric constraints

Using visual odometry and inertial measurements, indoor and outdoor positioning systems can perform an accurate self-localization in unknown, unstructured environments where absolute positioning systems (e.g. GNSS) are unavailable. However, the achievable accuracy is highly affected by the residuals of calibration, the quality of the noise model, etc. Only if these unavoidable uncertainties of sensors and data processing can be taken into account and be handled via error propagation, which allows to propagate them through the entire system. The central filter (e.g. Kalman filter) of the system can then make use of the enhanced statistical model and use the propagated errors to calculate the optimal result. In this paper, we focus on the uncertaintiy calculation of the elementary part of the optical navigation, the template feature matcher. First of all, we propose a method to model the image noise. Then we use Taylor’s theorem to extend two very popular and efficient template feature matchers sum-of-absolute-differences (SAD) and normalized-cross-correlation (NCC) to get sub-pixel matching results. Based on the proposed noise model and the extended matcher, we propagate the image noise to the uncertainties of sub-pixel matching results. Although the SAD and NCC are used, the image noise model can be easily combined with other feature matchers. We evaluate our method by an Integrated Positioning System (IPS) which is developed by German Aerospace Center. The experimental results show that our method can improve the quality of the measured trajectory. Moreover, it increases the robustness of the system.

In visual odometry applications, tracking of features in a video sequence greatly impacts the accuracy of ego-motion estimation. A robust visual tracking has to take into account either geometric or photometric conditions to exclude outliers in the stage of motion estimation. In this work we propose a novel method to adaptively constrain the matching of features in time sequence. The control of matching is based on two major Euclidean properties, namely scale and angle. The scale constraint refers to the disparity image, and the angular criterion is derived from epipolar geometry. The proposed method is tested with two selected sequences from KITTI visual odometry benchmark datasets. Experimental results indicate promising improvement is attainable over a conventional feature tracker for visual odometry.

This paper proposes a framework for tightly coupled monocular visual-inertial odometry (VIO) via epipolar constraints, which achieves accurate and effective 6 degrees-of-freedom (DOF) state estimation. After extracting and tracking of features, initial values of system state are then estimated for the VIO. The algorithm proposed establishes visual-inertial odometry atop a factor graph, which estimates the pose, velocity and bias of the IMU and the external between camera and IMU, allowing multiple measurements to be captured by different sensors as factors. The performance of the algorithm proposed is testified through the EuRoC datasets, and compared against other state-of-art VIO algorithms. The experimental results indicate that the introduction of epipolar constraints improves the state estimation accuracy effectively.

Dense stereo image matching in remotely sensed images is a challenging problem, though it has been studied for more than two decades, due to occlusions, discontinuities, geometric, and radiometric distortions. A novel multiobjective fitness function-based dense stereo matching approach using genetic algorithms (GAs) is proposed in this paper. The proposed method is useful for estimating dense disparity map with an improved number of inliers for a stereo image pair, despite the constraint of finding correct disparity at depth discontinuities. In this paper, the steps of GA, such as initialization of the population, fitness function, and crossover and mutation operation, are designed and implemented to effectively deal with the problem of dense stereo image matching. To initialize the population, a Scale Invariant Feature Transform (SIFT) descriptor is computed for each pixel and multiple-size window-based matching is performed, using the similarity measures: 1) Euclidean distance and 2) spectral angle mapper. The generated disparity maps are pruned to choose a suitable subset using the designed fitness functions, considering the constraints related to stereo image pair, such as epipolar constraint, which encodes the epipolar geometry and the similarity measure that is useful to decide accuracy of the correspondences. The two objective functions are the number of inliers computed using the fundamental matrix and an energy minimization function, considering discontinuities and occlusions. The usefulness of this approach for remotely sensed stereo image pairs is demonstrated by improving the number of inliers and favorably comparing with state-of-the-art dense stereo image matching methods.

Feature tracking is an efficient way to derive glacier velocity. It is based on a cross-correlation algorithm that seeks offsets of the maximal correlation windows on repeated satellite images. In this paper we demonstrate that different window sizes lead to different velocities. The averaged velocity gradient (AVG) method is proposed to improve window sizes in feature tracking and to obtain the most suitable flow field. The AVG method measures velocity variation between adjacent windows on the whole glacier in the image. Different window sizes lead to different AVG values, and the best-size window corresponds to the value where the AVG changes from abrupt to gradual. Using improved feature tracking, two flow fields of the same glacier are acquired with Advanced Land Observing Satellite (ALOS) optical and synthetic aperture radar (SAR) data, respectively. The advantages, application conditions, accuracy and disadvantages of the two kinds of data using the feature tracking method are discussed.

Nowadays UAVs are increasingly used to collect low altitude remote sensing images. But because of the instability of such flight platforms, collected images usually show some obvious disadvantages, such as the large scale difference, the large swing angle, with the result that the traditional matching methods are difficult to obtain satisfactory results of extracted feature points. That causes an immense obstacle in the feature matching between adjacent images. However, advantages of low altitude remote sensing are more appealing, which can provide very high spatial resolution images with more features and it is very useful especially in small area environmental monitoring. In fact, the gray-level-based operator of point-feature-based image matching method has been widely used in recent years for its rapidity, accuracy and stronger ability to resist deformation. In this paper, through contrast experiments, the authors compare the extracting performance among the three kinds of gray-level-based operators (the Forstner operator, the Harris operator and the SIFT operator) and find that the SIFT operator has a stable performance and the best property under various conditions which is nice to satisfy the functional requirement of the feature point extraction, with a strong practicability in the field of low altitude remote sensing image preprocessing. Besides, this paper puts forward a new Gauss Pyramid simplified model and descriptor generation method on the theory level and shows that the stability and timeless of the improved SIFT are better than the traditional algorithm through comparative experiment.

Spherical images have the advantage of recording full scenes using only one camera exposure and have been becoming an important data source for 3D reconstruction. However, geometric distortions inevitably exist due to the spherical camera imaging model. Thus, this study proposes a reliable feature matching algorithm for spherical images via the combination of local geometric rectification and CNN (convolutional neural network) learned descriptor. First, image patches around keypoints are reprojected to their corresponding tangent planes based on a spherical camera imaging model, which uses scale and orientation data from the keypoints to achieve both rotation and scale invariance. Second, feature descriptors are then calculated from the rectified image patches by using a pre-trained separate detector and descriptor learning network, which improves the discriminability by exploiting the high representation learning ability of the CNN. Finally, after classical feature matching with the ratio test and cross check, refined matches are obtained based on an essential matrix-based epipolar geometry constraint for outlier removal. By using three real spherical images and an incremental structure from motion (SfM) engine, the proposed algorithm is verified and compared in terms of feature matching and image orientation. The experiment results demonstrate that the geometric distortions can be efficiently reduced from rectified image patches, and the increased ratio of the match numbers ranges from 26.8% to 73.9%. For SfM-based spherical image orientation, the proposed algorithm provides reliable feature matches to achieve complete reconstruction with comparative accuracy.