Loop closure detection with patch-level local features and visual saliency prediction

Loop closure detection is the core part of the visual simultaneous localization and mapping (VSLAM) for autonomous robots. In the dynamic environments, loop closure detection turns to be a very difficult problem compared to the static scenario. This paper proposes a novel approach based on convolutional autoencoder neural network (CAENN) architecture for extracting images features, and then uses Euclidean loss for minimizing the difference between the extracted feature and the gist feature which has the ability of scene recognition. In order to improve the accuracy and recall rate for loop closure detection in the dynamic scenario, the perspective transformation and dynamic object are applied in the process of the construction of the training set. And by calculating the Manhattan distance between the two image feature vectors, the loop closure is accepted when the distance is smaller than the threshold. The experimental results demonstrate that the proposed method obtains better accuracy and recall rate compared to the commonly used gist feature method and has a lower cost on time and space compared to the BOW method and AlexNet method in loop closure detection.

Modest-vocabulary loop-closure detection with incremental bag of tracked words

Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.

Humans maintain good memory and recognition capability of previous environments when they are learning about new ones. Thus humans are able to continually learn and increase their experience. It is also obvious importance for autonomous mobile robot. The simultaneous localization and mapping system plays an important role in localization and navigation of robot. The loop-closure detection method is an indispensable part of the relocation and map construction, which is critical to correct mappoint errors of simultaneous localization and mapping. Existing visual loop-closure detection methods based on deep learning are not capable of continual learning in terms of cross-scene environment, which bring a great limitation to the application scope. In this article, we propose a novel end-to-end loop-closure detection method based on continual learning, which can effectively suppress the decline of the memory capability of simultaneous localization and mapping system by introducing firstly the orthogonal projection operator into the loop-closure detection to overcome the catastrophic forgetting problem of mobile robot in large-scale and multi-scene environments. Based on the three scenes from public data sets, the experimental results show that the proposed method has a strong capability of continual learning in the cross-scene environment where existing state-of-the-art methods fail.

Loop closure detection is a fundamental problem for visual simultaneous localization and mapping (VSLAM) in robotics. However, the current loop closure detection is mostly based on pixel-level recognition and matching algorithms which often fail under drastic viewpoint changes and illumination variations. This work is based on the idea that topological graph representation has better abstraction and globality for indoor scene. Based on this knowledge, we propose a method that pays more attention to scene global information to model visual scenes as semantic topological graphs by preserving only semantic information from object detection and geometric information from RGB-D cameras. We use the random walk method to traverse the graph structure to construct the graph descriptor implementing graph matching. Furthermore, the shape similarity and the Euclidean distance between objects in the 3D space are leveraged unitedly to measure the graph similarity. Comparing our method with existing classical methods in TUM dataset and indoor realistic complex scenes, the results show that our method has good performance compared to appearance-based and semantic-based methods.

With the emerging techniques of wireless communication and cloud computing, large volumes of multimedia data are outsourced from resource constrained users to the cloud with abundant resource for both delegated storage and computation. Unfortunately, there is a risk of users’ image privacy leakage in the process of outsourcing to untrusted cloud. Most of the existing work achieved privacy-preserving image feature extraction and matching by using public key (fully) homomorphic encryption (FHE), but the heavy computational overhead and communication overhead cannot adapt to resource-constrained mobile devices. Other works disabled to realize image denoising in the encrypted domain or only focused on the scale-invariant feature transform (SIFT) descriptor that is inappropriate for position-sensitive feature extraction. To address these issues, in this paper, a privacy-preserving shape context based image denoising and matching protocol PPOIM with efficient outsourcing is proposed. Firstly, to improve the accuracy of image matching, a privacy-preserving image denoising scheme PPID is proposed without exploiting public key FHE. Then, based on PPID, a privacy-preserving image matching protocol PPOIM adopting shape context descriptor is devised, where two secure and efficient comparison and counting protocols in the encrypted domain are presented. All the original image privacy, query image privacy and image matching result privacy are well protected. Finally, formal security proof and extensive simulations on real-world data sets demonstrate the efficiency and practicability of our proposed PPOIM.

Loop closure detection is essential for simultaneous localization and mapping systems to decrease accumulating drift of trajectory estimations. Robust loop closure detection is specifically important in large-scale mapping, but it gets more challenging as the mapping environment grows. This paper proposes a SLAM system utilizing a two-pass loop closure detection method to improve mapping accuracy in large-scale environments. The proposed system finds loop closure candidates by employing global and local image features together. After selecting a group of candidates by similarity of global features, the system applies keypoint matching on this group to improve scene matching accuracy and determines loop closure candidates. We extensively evaluate the system on the widely used TUM RGB-D dataset, which contains sequences of small to large-scale indoor environments, with respect to different parameter combinations. The results show that the proposed method increases accuracy substantially and achieves large-scale mapping with acceptable overhead.

A typical approach to describe an image in loop closure detection for visual SLAM is to extract a set of local patch descriptors and encode them into a co-occurrence vector. The most common patch encoding strategy is known as bag-of-visual-words (BoVW) representation, which consists of clustering the local descriptors into visual vocabulary. The distinctiveness of images is difficult to represent since most of them contain similar texture information, which may lead to false positive results. In this paper, the vocabulary is used as a whole by adopting the Fisher kernel (FK) framework. The new representation describes the image as the gradient vector of the likelihood function. The efficiently computed vectors can be compressed with a minimal loss of accuracy using product quantization and perform well in the task of loop closure detection. The proposed method achieves a higher recall rate with 100% precision in loop closure detection compared with state-of-the-art methods, and the detection on bidirectional loops is also enhanced. vSLAM systems may perceive the environment more efficiently by constructing a globally consistent map with the proposed loop closure detection method, which is potentially valuable for applications such as autonomous driving.KeywordsFisher kernelLoop closure detectionvisual SLAM

Place recognition of pre-visited areas, widely known as Loop Closure Detection (LCD), constitutes one of the most important components in robotic applications, where the robot needs to estimate its pose while navigating through the field (e.g., simultaneous localization and mapping). In this paper, we present a novel approach for LCD based on the assignment of Visual Words (VWs) to particular places of the traversed path. The system operates in real time and does not require any pre-training procedure, such as visual vocabulary construction or descriptor-space dimensionality reduction. A place is defined through a dynamic segmentation of the incoming image stream and is assigned with VWs through the usage of an on-line clustering algorithm. At query time, image descriptors are converted into VWs on the map accumulating votes to the corresponding places. By means of a probability function, the mechanism is capable of identifying a loop closing candidate place. A nearest neighbor voting scheme on the descriptors' space allows the system to select the most appropriate image match at the chosen place. Geometrical and temporal consistency checks are applied on the proposed loop closing pair increasing the system's performance. Evaluation took place on several publicly available and challenging datasets offering high precision and recall scores as compared to other state-of-the-art approaches.

This paper addresses the problem of constructing a globally consistent topological map by a vision-based mobile robot in an indoor environment. The vertices of the map represent the appearance of certain places and the edges represent the relative position between two vertices. An omnidirectional image matching algorithm is proposed to solve two problems: detecting loop closure and calibrating odometry feedback, which uses matched radial lines in successive frames to estimate the rotational angles of the robot. The map construction task consists of two steps: when no loop closure is detected, the robot use image matching results to calibrate the odometry data. After loop closure is detected, an energy-based spring model is used to refine the structure of the global map by minimize the energy of the model. Experimental evidence of the correctness of this method is given in unmodified indoor environments.

A fast method for mobile robot 3D SLAM (simultaneous localization and mapping) is presented to address the problem of 3D modeling in complex indoor environment. According to the camera calibration model and the image feature extraction and matching procedure, the association between two 3D point clouds can be established. On the basis of the RANSAC (random sample consensus) algorithm, the correspondence based Normal Distribution Transform algorithm (NDT) arithmetic model is solved to realize the robot's initial localization effectively. The loop detection method, which combines the space based method and appearance similarity method, promotes the speed of detection. Contrast experiments based on the actual scene show that, compared with the RGBD-SLAM and RTAB-MAP (realtime appearance-based mapping), the improved hybrid loop-closure detection algorithm has better real-time performance.

Loop closure detection is an indispensable part of the SLAM system, which can reduce the cumulative error of mobile robots in positioning and mapping. This paper proposes a supervised learning method, which combines the Siamese network and the VGG16 network in deep learning to handle loop closure detection. The Siamese network method and the traditional method are contrasted in the New College and City Centre data sets to prove the method’s usefulness in this research. The experiment’s findings reveal that the method of this paper ensures precision and recall rate while reducing the time necessary to assess similarities compared to the traditional method. It has a better chance of achieving real-time SLAM.

Loop closure detection (LCD) is crucial for the simultaneous localization and mapping system of an autonomous robot. Image features from a convolution neural network (CNN) have been widely used for LCD in recent years. Instead of directly using the feature vectors to compute the image similarity, we propose a novel and easy-to-implement method that manages features from a CNN via a novel approach to improve the performance. In this method, the elements of feature maps from the higher layer of the CNN are clustered to generate CNN words (CNNW). To encode spatial information of CNNW, we create word pairs (CNNWP) that are based on single words to improve the performance. In addition, traditional tricks that are used in methods that are based on bag of words (BoW) are integrated into our approach. We also demonstrate that the feature maps from lower layers can be used as descriptors to conduct local region matching between images. Via this approach, we can perform geometric verification for possible loop closures, similar to BoW methods, in our approach. The experimental results demonstrate that our method substantially outperforms state-of-the-art methods that directly use CNN features for LCD.

We address the problems of localization, mapping, and guidance for robots with limited computational resources by combining vision with the metrical information given by the robot odometry. We propose in this article a novel light and robust topometric simultaneous localization and mapping framework using appearance-based visual loop-closure detection enhanced with the odometry. The main advantage of this combination is that the odometry makes the loop-closure detection more accurate and reactive, while the loop-closure detection enables the long-term use of odometry for guidance by correcting the drift. The guidance approach is based on qualitative localization using vision and odometry, and is robust to visual sensor occlusions or changes in the scene. The resulting framework is incremental, real-time, and based on cheap sensors provided on many robots (a camera and odometry encoders). This approach is, moreover, particularly well suited for low-power robots as it is not dependent on the image processing frequency and latency, and thus it can be applied using remote processing. The algorithm has been validated on a Pioneer P3DX mobile robot in indoor environments, and its robustness is demonstrated experimentally for a large range of odometry noise levels.

Feature matching is a fundamental problem in many computer vision tasks. This paper proposes a novel effective framework for mismatch removal, named LOcality-guided Global-preserving Optimization (LOGO). To identify inliers from a putative matching set generated by feature descriptor similarity, we introduce a fixed-point progressive approach to optimize a graph-based objective, which represents a two-class assignment problem regarding an affinity matrix containing global structures. We introduce a strategy that a small initial set with a high inlier ratio exploits the topology of the affinity matrix to elicit other inliers based on their reliable geometry, which enhances the robustness to outliers. Geometrically, we provide a locality-guided matching strategy, i.e., using local topology consensus as a criterion to determine the initial set, thus expanding to yield the final feature matching set. In addition, we apply local affine transformations based on reference points to determine the local consensus and similarity scores of nodes and edges, ensuring the validity and generality for various scenarios including complex nonrigid transformations. Extensive experiments demonstrate the effectiveness and robustness of the proposed LOGO, which is competitive with the current state-of-the-art methods. It also exhibits favorable potential for high-level vision tasks, such as essential and fundamental matrix estimation, image registration and loop closure detection.