Robust Visual Place Recognition under Simultaneous Variations in Viewpoint and Appearance

Recently, the methods based on Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in visual place recognition. CNN is a class of multilayer perceptrons, but unlike common multilayer perceptrons that it is not usually fully connected networks. It can acquire more general image features and make the image processing computationally manageable through filtering the connections by proximity. In this paper, we utilize the deep features generated by CNNs and the dynamic time warping (DTW) algorithm for image sequence place recognition. We propose a novel image similarity measurement, which is derived from cosine distance and can better distinguish match and mismatch. Meanwhile, we improve the DTW algorithm to design a local matching method that can reduce time complexity from O(n3) to O(n). To test the proposed method, four datasets (Nordland, Gardens Point, St. Lucia, and UoA datasets) are used as benchmarks; using two traverses in each dataset with one for reference and the other for testing. The results show high precision-recall characteristics of our method in the cases of severe appearance changes. Besides, our method achieves substantial improvements over the methods using the deep feature representations of a single image for recognition, which reflects that the spatiotemporal information contained in the image sequence is significant for the task of visual place recognition. Moreover, the proposed method also shows to outperform the classical sequence-based method SeqSLAM.

Visual Place Recognition (VPR) is a fundamental yet challenging task in Visual Simultaneous Localization and Mapping (V-SLAM) problems. The VPR works as a subsystem of the V-SLAM. VPR is the task of retrieving images upon revisiting the same place in different conditions. The problem is even more difficult for agricultural and all-terrain autonomous mobile robots that work in different scenarios and weather conditions. Over the last few years, many state-of-the-art methods have been proposed to solve the limitations of existing VPR techniques. VPR using bag-of-words obtained from local features works well for a large-scale image retrieval problem. However, the aggregation of local features arbitrarily produces a large bag-of-words vector database, limits the capability of efficient feature learning, and aggregation and querying of candidate images. Moreover, aggregating arbitrary features is inefficient as not all local features equally contribute to long-term place recognition tasks. Therefore, a novel VPR architecture is proposed suitable for efficient place recognition with semantically meaningful local features and their 3D geometrical verifications. The proposed end-to-end architecture is fueled by a deep neural network, a bag-of-words database, and 3D geometrical verification for place recognition. This method is aware of meaningful and informative features of images for better scene understanding. Later, 3D geometrical information from the corresponding meaningful features is computed and utilised for verifying correct place recognition. The proposed method is tested on four well-known public datasets, and Micro Aerial Vehicle (MAV) recorded dataset for experimental validation from Victoria Park, Adelaide, Australia. The extensive experimental results considering standard evaluation metrics for VPR show that the proposed method produces superior performance than the available state-of-the-art methods.

Visual Place Recognition (VPR), the task of identifying the place where an image has been taken from, is at the core of important robotic problems as relocalization, loop-closure detection or topological navigation. Even for indoors, the focus of this work, VPR is challenging for a number of reasons, including real-time performance when dealing with large image databases ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim10^{4}$</tex-math></inline-formula> ) (probably captured by different robots), or the avoidance of Perceptual Aliasing in environments with repetitive structures and scenes. In this letter, we tackle these issues by proposing an off-line mapping technique that abstracts a dense database of georeferenced images without particular order into a Multivariate Gaussian Mixture Model, by creating soft clusters in terms of their similarity in both pose and appearance. This abstract representation is obtained through an Expectation-Maximization algorithm and plays the role of a simplified map. Since querying this map yields a probability of being in a cluster, we exploit this “belief” within a Bayesian filter that regards previous query images and a topological map between clusters to perform more robust VPR. We evaluate our proposal in two different indoor datasets, demonstrating comparable VPR precision to querying the full database while incurring in shorter query times and handling Perceptual Aliasing for sequential navigation.

Appearance-invariant place recognition by discriminatively training a convolutional neural network

Visual place recognition: A survey from deep learning perspective

A distinctive feature of intelligent systems is their capability to analyze their level of expertise for a given task; in other words, they know what they know. As a way towards this ambitious goal, this paper presents a recognition algorithm able to measure its own level of confidence and, in case of uncertainty, to seek for extra information so to increase its own knowledge and ultimately achieve better performance. We focus on the visual place recognition problem for topological localization, and we take an SVM approach. We propose a new method for measuring the confidence level of the classification output, based on the distance of a test image and the average distance of training vectors. This method is combined with a discriminative accumulation scheme for cue integration. We show with extensive experiments that the resulting algorithm achieves better performances for two visual cues than the classic single cue SVM on the same task, while minimising the computational load. More important, our method provides a reliable measure of the level of confidence of the decision.

Visual place recognition is a fundamental problem for many vision based applications. Sparse feature and deep learning based methods have been successful and dominant over the decade. However, most of them do not explicitly leverage high-level semantic information to deal with challenging scenarios where they may fail. This paper proposes a novel visual place recognition algorithm, termed TextPlace, based on scene texts in the wild. Since scene texts are high-level information invariant to illumination changes and very distinct for different places when considering spatial correlation, it is beneficial for visual place recognition tasks under extreme appearance changes and perceptual aliasing. It also takes spatial-temporal dependence between scene texts into account for topological localization. Extensive experiments show that TextPlace achieves state-of-the-art performance, verifying the effectiveness of using high-level scene texts for robust visual place recognition in urban areas.

Omnidirectional cameras are a suitable and cost-effective choice for Visual Place Recognition (VPR), as they provide comprehensive information from the scene regardless of the robot orientation. However, vision sensors are vulnerable to environmental appearance changes (e.g., illumination, weather, season or moving objects). While multi-modal sensing approaches can overcome these challenges, they introduce significant cost and system complexity. This paper introduces PDPR (Panoramic-Depth Place Recognition), a novel fusion framework that enhances the robustness of VPR methods by integrating visual data with geometric features derived from monocular depth estimation techniques, while using a single-camera setup. In the ablation study, both early and late fusion strategies are evaluated to optimally combine appearance-based and depth-derived features. The extensive evaluation on challenging, indoor and outdoor datasets demonstrates that PDPR consistently boosts retrieval performance across multiple state-of-the-art VPR models. Furthermore, this improvement is achieved without requiring any fine tuning, allowing our method to function as a pluggable module for pretrained models. Consequently, this work presents a powerful, practical and low-cost solution for robust VPR, with high potential to scale as monocular depth estimation and VPR models continue to improve. The project website can be found at https://marcosalfaro.github.io/projects-PDPR/ . • Monocular depth estimation is used to enhance place recognition. • A thorough evaluation of preprocessing techniques to enhance the depth maps. • Fusion techniques are designed to leverage visual and geometric data. • A model-agnostic approach that improves the performance even with no fine tuning. • A robust method across different scenarios and lighting conditions.

Visual Place Recognition (VPR) is process of properly identifying a formerly visited place under varying viewpoints conditions. VPR becomes a challenging task due to the variations in lighting conditions (day time or night time), shadows, weather conditions, view points, or even different seasons. Therefore, it is needed to develop VPR algorithms which can handle the variation in visual appearances. This article designs a visual place recognition using a novel region of interest (RoI) extraction with deep learning based (VPR-ROIDL) technique on changing environments. The proposed VPR-ROIDL technique initially enables the extraction of RoIs using saliency map then features of RoIs using local diagonal extrema patterns (LDEP) respectively. Besides, the extracted features of RoIs are passed into the convolutional neural network based residual network (ResNet) model and the computed deep features are stored in the database. Upon providing a query image (QI), the RoIs and features of RoIs are extracted as same as reference images. Moreover, the deep features from the ResNet model are generated and Euclidean distance based similarity measurement is used to finally recognize the places. By the use of RoI and descriptors matching process, the VPR-ROIDL model has the ability to recognize places irrespective of changing illuminations, seasons, and viewpoints. The simulation results pointed out the supremacy of the ROIDL-VPR technique over the recent state of art approaches.KeywordsVisual Place recognitionDeep learningVisual appearancesRegion of InterestSaliency mapSimilarity Measurement

Visual Place Recognition (VPR) is essential for robotics and autonomous navigation, yet most methods rely on heavy task-specific training. Existing approaches fall into two main paradigms: single-stage models that learn compact global descriptors, and two-stage pipelines that combine coarse global retrieval with local feature or geometric verification. While effective, both require large annotated datasets and carefully tuned optimization, limiting scalability and cross-domain reuse. We introduce TF-VPR, a new benchmark that tackles a more challenging setting: VPR performed entirely without additional training, where descriptors are generated, refined and matched only at test time. Enabled by recent Vision Foundation Models (VFMs), TF-VPR systematically evaluates how far pretrained VFMs can be pushed for place recognition when used as-is, and provides a standardized protocol for fairly comparing arbitrary VFMs without fine-tuning. To support this, we unify major VPR datasets covering diverse real-world conditions and propose two lightweight, training-free modules: Training-Free Graph-Attention Graph Module (TF-GAM) and Training-Free Cross-Attention Module (TF-CAM). These plug-and-play modules enhance descriptor discriminability and retrieval robustness. Experiments show that TF-VPR exposes new challenges and reveals previously unexplored strengths of VFMs for training-free place recognition. Code and datasets are available at https://github.com/ddfs430/TF-VPR .

Visual place recognition (VPR) is critical in not only localization and mapping for autonomous driving vehicles, but also assistive navigation for the visually impaired population. To enable a long-term VPR system on a large scale, several challenges need to be addressed. First, different applications could require different image view directions, such as front views for self-driving cars while side views for the low vision people. Second, VPR in metropolitan scenes can often cause privacy concerns due to the imaging of pedestrian and vehicle identity information, calling for the need for data anonymization before VPR queries and database construction. Both factors could lead to VPR performance variations that are not well understood yet. To study their influences, we present the NYU-VPR dataset that contains more than 200,000 images over a 2km×2km area near the New York University campus, taken within the whole year of 2016. We present benchmark results on several popular VPR algorithms showing that side views are significantly more challenging for current VPR methods while the influence of data anonymization is almost negligible, together with our hypothetical explanations and in-depth analysis.

Visual Place Recognition (VPR) in areas with similar scenes such as urban or indoor scenarios is a major challenge. Existing VPR methods using global descriptors have difficulty capturing local specific region (LSR) in the scene and are therefore prone to localization confusion in such scenarios. As a result, finding the LSRs that are critical for location recognition becomes key. To address this challenge, we introduced Patch-NetVLAD+, which was inspired by patch-based VPR researches. Our method proposed a fine-tuning strategy with triplet loss to make NetVLAD suitable for extracting patch-level descriptors. Moreover, unlike existing methods that treat all patches in an image equally, our method extracts patches of LSR, which present less frequently throughout the dataset, and makes them play an important role in VPR by assigning proper weights to them. Experiments on Pittsburgh30k and Tokyo247 datasets show that our approach achieved up to 9.3% performance improvement than existing patch-based methods.

Event cameras are neuromorphic vision sensors that are able to capture high dynamic range with low latency in microseconds, without motion blur. Their strength lies in the unique representation of data as asynchronous events, enabling detection of scene structures less invariantly from dynamic luminance changes. However, a single event does not represent spatial information, and events must be integrated to translate into meaningful information. Therefore, state-of-the-art deep learning algorithms have focused on reconstructing the original scene from events. However, as environmental variances are also captured throughout events and restored in reconstructed images, simple reconstruction does not help achieving robust visual place recognition. In this paper, we suggest to use reconstructed event edges denoised for place recognition. While brightness wavers with dynamic environmental variances, edge contours only change with gradient magnitude scale. We utilize the high dynamic range of event cameras to detect these scaled edges from different environments and show that using reconstructed edges shows robust performance in overcoming day-to-night illumination variance without a large training set.

Visual place recognition is a mature field in mobile robotics research. Recognizing places in datasets covering traversals of hundreds or thousands of kilometres and accurate localization in small and medium size environments have been successfully demonstrated. However, for real world long term operation, visual place recognition has to face severe environmental appearance changes due to day-night cycles, seasonal or weather changes. Existing approaches for recognizing places in such changing environments provide solutions for matching images from the exact same viewpoint using powerful holistic descriptors, using less sophisticated holistic descriptors in combinations with images sequences, and/or pose strong requirements on training data to learn systematic appearance changes. In this paper, we present a novel, training free, single image matching procedure that builds upon local region detectors for powerful Convolutional Neural Network (CNN) based descriptors. It can be used with a broad range of local region detectors including keypoints, segmentation based approaches and object proposals. We propose a novel evaluation criterion for selection of an appropriate local region detector for changing environments and compare several available detectors. The scale space extrema detector known from the SIFT keypoint detector in combination with appropriate magnification factors performs best. We present preliminary results of the proposed image matching procedure with several region detectors on the challenging Nordland dataset on place recognition between different seasons and a dataset including severe viewpoint changes. The proposed method outperforms the best existing holistic method for place recognition in such changing environments and can additionally handle severe viewpoint changes. Additionally, the combination of the best performing detectors with superpixel based spatial image support shows promising results.

Traditional visual place recognition (VPR), usually using standard cameras, is easy to fail due to glare or high-speed motion. By contrast, event cameras have the advantages of low latency, high temporal resolution, and high dynamic range, which can deal with the above issues. Nevertheless, event cameras are prone to failure in motionless scenes, while standard cameras can still provide appearance information in this case. Thus, exploiting the complementarity of standard cameras and event cameras can effectively improve the performance of VPR algorithms. In the paper, we propose FE-Fusion-VPR, an attention-based multi-scale network architecture for VPR by fusing frames and events. First, the intensity frame and event volume are fed into the two-stream feature extraction network for shallow feature fusion. Next, the three-scale features are obtained through the multi-scale fusion network and aggregated into three sub-descriptors using the VLAD layer. Finally, the weight of each sub-descriptor is learned through the descriptor re-weighting network to obtain the final refined descriptor. Experimental results show that our FE-Fusion-VPR outperforms existing frame-based, event-based and fusion-based VPR methods in most cases on Brisbane-Event-VPR and DDD20 datasets. In a word, compared to the previous works, our FE-Fusion-VPR achieves new state-of-the-art (SOTA) VPR performance on Brisbane-Event-VPR and DDD20 datasets by fusing frames and events.