CSFD-AFM: Multimodal Remote Sensing Image Registration via Cyclic Shift Feature Descriptor and Adaptive Feature Matching Strategy

Co-registration of multimodal remote sensing images (MRSI) remains a challenging task due to nonlinear radiometric differences (NRD) and geometric distortions. This letter proposes a robust registration method based on steerable Hermite filters and geometric constraints. The method consists of three stages: coarse registration, image matching, and fine registration. In the first stage, geo-referencing information is used to partition the overlapping region into sub-blocks for initial alignment. In the second stage, steerable Hermite filters are applied to extract multi-scale and multi-orientation structural features, which are fused into a principal moment map to detect highly repeatable feature points. A structural descriptor is subsequently constructed from an orientation index map generated through hierarchical pooling, enabling robust matching under NRD. In the final stage, an iterative optimization strategy incorporating geometric constraints is employed to eliminate mismatches and refine the registration. Experimental results on multiple MRSI datasets demonstrate that the proposed method outperforms state-of-the-art approaches in terms of both matching robustness and registration accuracy.

Many computer vision applications rely on feature matching, hence the need for computationally efficient and robust 4D light field (LF) feature detectors and descriptors for applications using this imaging modality. In this paper, we propose a novel LF feature extraction method in the scale-disparity space, based on a Fourier disparity layer representation. The proposed feature extraction takes advantage of both the Harris feature detector and SIFT descriptor, and is shown to yield more accurate feature matching, compared with the LiFF light field feature with low computational complexity. In order to evaluate the feature matching performance with the proposed descriptor, we generated synthetic LF datasets with ground truth matching points. Experimental results with synthetic and real datasets show that, our solution outperforms existing methods in terms of both feature detection robustness and feature matching accuracy.

Accurate and robust day-night feature matching is a fundamental challenge for visual localization of autonomous driving cars. Under adverse illumination changing situations, the performance of current handcrafted and CNN-based local features will degrade severely. This problem is caused by the phenomenon of domain gap, which can be alleviated by applying domain transformation to increase the discriminativeness of feature descriptors. However, the standard optimization function of descriptors with hard negative triplets will lead to local minima, which will significantly decrease the robustness of domain transformation on feature matching. To address above challenges, the absolute anchor-negative distance based loss function is proposed, which is named AAN loss. The proposed AAN loss integrates the absolute distance of anchor-negative and anchor-positive samples, to effectively strengthen the discriminativeness of descriptors and prevent converging to local minima. Our proposed method can improve the convergence of the domain transformation and effectively improve the performance of feature matching under adverse illumination conditions. Extensive experiments and evaluations show the improved robustness and efficiency of the proposed method.

Image-based localization plays an important role in today's autonomous driving technologies. However, in large scale outdoor environments, challenging conditions, e.g., lighting changes or different weather, heavily affect image appearance and quality. As a key component of feature-based visual localization, image feature detection and matching deteriorate severely and cause worse localization performance. In this paper, we propose a novel method for robust image feature matching under drastically changing outdoor environments. In contrast to existing approaches which try to learn robust feature descriptors, we train a deep network that outputs the low-rank representations of the images where the undesired variations on the images are removed, and perform feature extraction and matching on the learned low-rank space. We demonstrate that our learned low-rank images largely improve the performance of image feature matching under varying conditions over a long period of time.

A feature descriptor that is robust to a number of image deformations is a basic requirement for vision based applications. Most feature descriptors work well in image deformations such as compression artifacts, illumination changes, and blurring. To develop a feature descriptor that works well apart from these image deformations like transformations caused by long baseline is a challenging task. This paper introduces a compact and efficient binary feature descriptor called PRObabilistic (PRO). A method for removing non-affine features from the initial feature list is developed, which results in further improved performance with the PRO descriptor when dealing with many deformations including long baseline between images. Feature matching accuracy using only affine features is compared with accuracy using both affine and non-affine features on benchmark datasets to demonstrate the advantages of using affine feature point for PRO descriptor.

Spherical images have the advantage of recording full scenes using only one camera exposure and have been becoming an important data source for 3D reconstruction. However, geometric distortions inevitably exist due to the spherical camera imaging model. Thus, this study proposes a reliable feature matching algorithm for spherical images via the combination of local geometric rectification and CNN (convolutional neural network) learned descriptor. First, image patches around keypoints are reprojected to their corresponding tangent planes based on a spherical camera imaging model, which uses scale and orientation data from the keypoints to achieve both rotation and scale invariance. Second, feature descriptors are then calculated from the rectified image patches by using a pre-trained separate detector and descriptor learning network, which improves the discriminability by exploiting the high representation learning ability of the CNN. Finally, after classical feature matching with the ratio test and cross check, refined matches are obtained based on an essential matrix-based epipolar geometry constraint for outlier removal. By using three real spherical images and an incremental structure from motion (SfM) engine, the proposed algorithm is verified and compared in terms of feature matching and image orientation. The experiment results demonstrate that the geometric distortions can be efficiently reduced from rectified image patches, and the increased ratio of the match numbers ranges from 26.8% to 73.9%. For SfM-based spherical image orientation, the proposed algorithm provides reliable feature matches to achieve complete reconstruction with comparative accuracy.

Due to significant nonlinear radiometric differences (NRD) and severe geometric distortions (e.g., translation, scale, and rotation) among multimodal images (e.g., optical, infrared, SAR), image registration remains a challenging problem. This paper proposes a robust matching method based on self-similarity adjacent self-convolution features (SASF), consisting of two key steps. In the first stage, a uniform distribution of feature points in multimodal images is extracted using a FAST feature extraction operator based on normalized filtering, and geometric correction is performed using prior reference geographic location information of remote sensing images to achieve coarse registration. In the second stage, addressing the significant NRD, a novel structural feature SASF is constructed. Based on this, a rapid structural similarity measurement method (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Fast-SSIM_{SASF}$</tex-math></inline-formula>) is proposed, which utilizes 3D fast Fourier transform techniques to enhance image matching efficiency. The proposed SASF algorithm is evaluated for performance on multiple sets of multimodal remote sensing images. Experimental results demonstrate that this method outperforms many existing excellent algorithms in terms of matching effectiveness. Finally, the SASF algorithm is tested on four pairs of entire-scene multimodal remote sensing satellite images, demonstrating its robustness to nonlinear radiometric differences and geometric distortions.

In 3D object recognition, local feature-based recognition is known to be robust against occlusion and clutter. Local feature estimation requires feature correspondences, including feature extraction and matching. Feature extraction is normally a two-stage process that estimates keypoints and keypoint descriptors, and existing studies show repeatability to be a good indicator of keypoint feature detector robustness. However, the impact of keypoint repeatability on feature correspondence estimation and overall feature matching accuracy has not yet been studied. In this paper, local features are extracted at both regular and repeatable 3D keypoints using leading keypoint detectors combined with the SHOT descriptor to estimate a set of correspondences. When using a keypoint detector of high repeatability, experimental results show improved feature matching accuracy and reduced computational requirements for the feature description and matching, and overall correspondence estimation process.

Abstract. Feature detection and matching are key techniques in computer vision and robotics, and have been successfully implemented in many fields. So far there is no performance comparison of feature detectors and matching methods for planetary mapping and rover localization using rover stereo images. In this research, we present a comprehensive evaluation and comparison of six feature detectors, including Moravec, Förstner, Harris, FAST, SIFT and SURF, aiming for optimal implementation of feature-based matching in planetary surface environment. To facilitate quantitative analysis, a series of evaluation criteria, including distribution evenness of matched points, coverage of detected points, and feature matching accuracy, are developed in the research. In order to perform exhaustive evaluation, stereo images, simulated under different baseline, pitch angle, and interval of adjacent rover locations, are taken as experimental data source. The comparison results show that SIFT offers the best overall performance, especially it is less sensitive to changes of image taken at adjacent locations.

This work presents a visual information fusion approach for robust probability-oriented feature matching. It is sustained by omnidirectional imaging, and it is tested in a visual localization framework, in mobile robotics. General visual localization methods have been extensively studied and optimized in terms of performance. However, one of the main threats that jeopardizes the final estimation is the presence of outliers. In this paper, we present several contributions to deal with that issue. First, 3D information data, associated with SURF (Speeded-Up Robust Feature) points detected on the images, is inferred under the Bayesian framework established by Gaussian processes (GPs). Such information represents a probability distribution for the feature points’ existence, which is successively fused and updated throughout the robot’s poses. Secondly, this distribution can be properly sampled and projected onto the next 2D image frame in , by means of a filter-motion prediction. This strategy permits obtaining relevant areas in the image reference system, from which probable matches could be detected, in terms of the accumulated probability of feature existence. This approach entails an adaptive probability-oriented matching search, which accounts for significant areas of the image, but it also considers unseen parts of the scene, thanks to an internal modulation of the probability distribution domain, computed in terms of the current uncertainty of the system. The main outcomes confirm a robust feature matching, which permits producing consistent localization estimates, aided by the odometer’s prior to estimate the scale factor. Publicly available datasets have been used to validate the design and operation of the approach. Moreover, the proposal has been compared, firstly with a standard feature matching and secondly with a localization method, based on an inverse depth parametrization. The results confirm the validity of the approach in terms of feature matching, localization accuracy, and time consumption.

There has been significant research on the development of feature descriptors in the past few years. Most of them do not emphasize real-time applications. This paper presents the development of an affine invariant feature descriptor for low resource applications such as UAV and UGV that are equipped with an embedded system with a small microprocessor, a field programmable gate array (FPGA), or a smart phone device. UAV and UGV have proven suitable for many promising applications such as unknown environment exploration, search and rescue operations. These applications required on board image processing for obstacle detection, avoidance and navigation. All these real-time vision applications require a camera to grab images and match features using a feature descriptor. A good feature descriptor will uniquely describe a feature point thus allowing it to be correctly identified and matched with its corresponding feature point in another image. A few feature description algorithms are available for a resource limited system. They either require too much of the device’s resource or too much simplification on the algorithm, which results in reduction in performance. This research is aimed at meeting the needs of these systems without sacrificing accuracy. This paper introduces a new feature descriptor called PRObabilistic model (PRO) for UGV navigation applications. It is a compact and efficient binary descriptor that is hardware-friendly and easy for implementation.

During the continual transitions from lab research to real-world applications of vision-based algorithms, there are significant challenges, e.g. the robustness to adapt to complex environments and the high demands of multi-task, multi-modal learning models. More specifically on visual simultaneous localization and mapping (visual SLAM) for mobile robots, two notable limitations are: (i) the drift issue during pose estimation, especially in dynamic environments, makes the positioning system unstable; (ii) the additional loop detection takes many computational resources for image retrieval and global feature geometry checking. This thesis explores the approaches of visual localization on unmanned ground vehicles by solving those limitations. The conventional feature extraction and matching pipeline for vision-based tasks involve three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. The first contribution of the thesis is adding a feature selection and anticipation stage to reduce tracking drift. We explore the noise model of image features to select a subset of all the observed image features with the best ”contribution” during data association and pose estimation across multiple frames. Conventional SLAM algorithms take a strong assumption of scene rigidity, which limits the application under challenging environments. The second part of the thesis addresses the tough issue in dynamic environments with moving objects. We presented GMC, namely the motion clustering approach, a lightweight dynamic object filtering method. It can distinguish moving objects from static landmarks. Based on the theory of motion coherence within a particular image area, GMC could segment dynamic objects in 3D space. We can provide an efficient and robust correspondence algorithm that can extract dynamic objects from a static background with the method. In this way, we propose a dynamic SLAM system that is real-time and free from expensive GPU processors. In contrast to GMC, the thesis’s third part turns to an object-aware learning-based model for more general dynamic scenarios. We use object detection and tracking as points, lines, planes, etc. We utilize semantic information and extract sparse image features simultaneously to keep track of dynamic objects. The static background and different dynamic objects are jointly optimized in a newly developed bundle adjustment sliding window. The estimated 3D bounding boxes can provide more robust camera tracking and better scene understanding, and better map merging. The fourth part of the thesis leverages the emerging feature learning framework. It proposes a unified self-supervised model called LGDNet to generate both global and local image feature descriptors end-to-end. Global feature descriptors embed the whole image into a compact representation, leading to easier scene comparison. On the other hand, local features focus more on the local region similarities of some salient parts for structure from motion. Our proposed method can directly extract features together with descriptors that encode both local maximum responses and global context information, avoiding duplicate calculations based on different feature extraction criteria.

Grasping has long been studied in the field of robotics. In this paper, we divide the process of robotic grasp into sensing and control. In terms of sensing, 2D vision based sensing relies on accurate feature matching and object surface texture features, resulting in poor performance in the complex environment with occlusion. By contrast, some sensors based on 3D vision are more robust to noise. Processing point clouds in a deep learning method can achieve high accuracy as well as reducing the computation time compared with those using cost volume regularization. For the control part, the traditional trajectory motion methods are limited to generalization and grasping with high degrees of freedom. On the contrary, the methods of reinforcement learning can improve the grasping strategy in the continuous interaction with the environment. We propose some commonly used benchmarks and simulation platforms for simulation experiment using reinforcement learning.

Image registration is an indispensable component of remote sensing applications, such as disaster monitoring, change detection, and classification. Grayscale differences and geometric distortions often occur among multisource images due to their different imaging mechanisms, thus making it difficult to acquire feature points and match corresponding points. This article proposes a scene shape similarity feature (SSSF) descriptor based on scene shape features and shape context algorithms. A new similarity measure called SSSFncc is then defined by computing the normalized correlation coefficient of the SSSF descriptors between multisource remote sensing images. Furthermore, the tie points between the reference and the sensed image are extracted via a template matching strategy. A global consistency check method is then used to remove the mismatched tie points. Finally, a piecewise linear transform model is selected to rectify the remote sensing image. The proposed SSSFncc aims to extract the scene shape similarity between multisource images. The accuracy of the proposed SSSFncc is evaluated using five pairs of experimental images from optical, synthetic aperture radar, and map data. Registration results demonstrate that the SSSFncc similarity measure is robust enough for complex nonlinear grayscale differences among multisource remote sensing images. The proposed method achieves more reliable registration outcomes compared with other popular methods.

As the capsule endoscope lens will produce geometric distortion, while there is an angle and a corner between the camera image plane and the plane of scene, so the images of capsule endoscope get nonlinear geometric distortion. Before the capsule endoscope images are used to accurately analysis, we should eliminate these distortions. Typically, the correction uses a calibration pattern, which can be a chess board, dot, grid, or circle pattern. Based on this pattern, enough characteristic samples can be extracted accurately and conveniently, and mathematic model can be built for the distortion in the captured image with respect to the original calibration pattern. However, those methods are a kind of correction method using nonlinear function in all. For the case of local distortion situations, it is difficult to achieve the desired correction. This paper presents an image distortion correction method based on look-up table method. The method is to process image into blocks, making the pixels of the region on the image corrected. The experimental results show that: this method is reliable and effective.