Accurate Image Localization Based on Google Maps Street View

Change detection, or anomaly detection, from street-view images acquired by an autonomous robot at multiple different times, is a major problem in robotic mapping and autonomous driving. Formulation as an image comparison task, which operates on a given pair of query and reference images is common to many existing approaches to this problem. Unfortunately, providing relevant reference images is not straightforward. In this paper, we propose a novel formulation for change detection, termed compressive change retrieval, which can operate on a query image and similar reference images retrieved from the web. Compared to previous formulations, there are two sources of difficulty. First, the retrieved reference images may frequently contain non-relevant reference images, because even state-of-the-art place-recognition techniques suffer from retrieval noise. Second, image comparison needs to be conducted in a compressed domain to minimize the storage cost of large collections of street-view images. To address the above issues, we also present a practical change detection algorithm that uses compressed bag-of-words (BoW) image representation as a scalable solution. The results of experiments conducted on a practical change detection task, "moving object detection (MOD)," using the publicly available Malaga dataset validate the effectiveness of the proposed approach.

This paper presents a new approach for the challenging problem of geo-localization using image matching in a structured database of city-wide reference images with known GPS coordinates. We cast the geo-localization as a clustering problem of local image features. Akin to existing approaches to the problem, our framework builds on low-level features which allow local matching between images. For each local feature in the query image, we find its approximate nearest neighbors in the reference set. Next, we cluster the features from reference images using Dominant Set clustering, which affords several advantages over existing approaches. First, it permits variable number of nodes in the cluster, which we use to dynamically select the number of nearest neighbors for each query feature based on its discrimination value. Second, this approach is several orders of magnitude faster than existing approaches. Thus, we obtain multiple clusters (different local maximizers) and obtain a robust final solution to the problem using multiple weak solutions through constrained Dominant Set clustering on global image features, where we enforce the constraint that the query image must be included in the cluster. This second level of clustering also bypasses heuristic approaches to voting and selecting the reference image that matches to the query. We evaluate the proposed framework on an existing dataset of 102k street view images as well as a new larger dataset of 300k images, and show that it outperforms the state-of-the-art by 20 and 7 percent, respectively, on the two datasets.

Motivated by the recent success of supervised and weakly supervised common object discovery, in this paper, we move forward one step further to tackle common object discovery in a fully unsupervised way. Generally, object co-localization aims at simultaneously localizing objects of the same class across a group of images. Traditional object localization/detection usually trains specific object detectors which require bounding box annotations of object instances, or at least image-level labels to indicate the presence/absence of objects in an image. Given a collection of images without any annotations, our proposed fully unsupervised method is to simultaneously discover images that contain common objects and also localize common objects in corresponding images. Without requiring to know the total number of common objects, we formulate this unsupervised object discovery as a sub-graph mining problem from a weighted graph of object proposals, where nodes correspond to object proposals, and edges represent the similarities between neighbouring proposals. The positive images and common objects are jointly discovered by finding sub-graphs of strongly connected nodes, with each sub-graph capturing one object pattern. The optimization problem can be efficiently solved by our proposed maximal-flow-based algorithm. Instead of assuming that each image contains only one common object, our proposed solution can better address wild images where each image may contain multiple common objects or even no common object. Moreover, our proposed method can be easily tailored to the task of image retrieval in which the nodes correspond to the similarity between query and reference images. Extensive experiments on PASCAL VOC 2007 and Object Discovery data sets demonstrate that even without any supervision, our approach can discover/localize common objects of various classes in the presence of scale, view point, appearance variation, and partial occlusions. We also conduct broad experiments on image retrieval benchmarks, Holidays and Oxford5k data sets, to show that our proposed method, which considers both the similarity between query and reference images and also similarities among reference images, can help to improve the retrieval results significantly.

A method for ground control point (GCP) acquisition using a simulated synthetic aperture radar (SAR) image derived from a digital elevation model (DEM) is proposed. Also proposed is a method for the evaluation of the accuracy of GCP matching with texture features from a reference GCP chip image. Results from experiments with simulated GCP chip images as reference images and geometrically distorted GCP chip images, derived using simulated SAR images as current images, show good coincidence with GCP matching accuracy in terms of pixel distances between matched GCP points in reference and current chip images and texture features. In particular, the correlation coefficient between the angular second moment (ASM) and the matching accuracy is 0.757, followed by Ent (entropy), Horn (homoginiety), Dis (dissimilarity), Con (contrast), Var (variability coefficient) and Chi (Chi-square), for the skewed images while for the rotated images, Dis shows the highest correlation of 0.628, followed by Horn, Asm, Co...

A novel method for line matching is proposed. The basic idea is to use tentative point correspondences, which can be easily obtained by keypoint matching methods, to significantly improve line matching performance, even when the point correspondences are severely contaminated by outliers. When matching a pair of image lines, a group of corresponding points that may be coplanar with these lines in 3D space is firstly obtained from all corresponding image points in the local neighborhoods of these lines. Then given such a group of corresponding points, the similarity between this pair of lines is calculated based on an affine invariant from one line and two points. The similarity is defined on the basis of median statistic in order to handle the problem of inevitable incorrect correspondences in the group of point correspondences. Furthermore, the relationship of rotation between the reference and query images is estimated from all corresponding points to filter out those pairs of lines which are obviously impossible to be matches, hence speeding up the matching process as well as further improving its robustness. Extensive experiments on real images demonstrate the good performance of the proposed method as well as its superiority to the state-of-the-art methods.

The ‘‘self-comparison method’’ is used to measure positioning errors, thermal and mechanical instabilities as well as field distortions of a high-resolution electron beam lithography system. For drift measurements, line gratings are overlaid with slightly rotated gratings of the same constant in subsequent exposures. From the position of the moiré patterns we deduce positioning errors with 2.5 nm resolution. Since moiré measurements are equivocal, we use in addition vernier patterns with a resolution of 150 nm to overcome this problem. The measurements are made with an optical microscope but not with a calibrated metrology tool. Distortion measurement is carried out using an image acquisition system with 512×512 pixels together with a video camera and an optical microscope. Images of a point grating filling a full field are taken at various positions in the field. The grating in the center region is taken as a reference for the measurement. Field distortions are measured from differences in the grating constant and rotation between the center region and the outer regions. These are determined by comparing the center of mass of each grating point in the images with the corresponding points in the reference image. Differences in the grating constant down to 0.0003 times the constant and rotations of 0.0003 rad can be measured. These results are then fitted to a polynomial function of fifth order representing the field aberrations. Using this method, the distortions of the image acquisition system have no influence on the result of the evaluations .

In this paper, we propose an effective time-of-arrival (TOA)-based localization method with adaptive bias computation in indoor environments. The goal of the localization is to estimate an accurate target's location in wireless localization system. However, in indoor environments, non-line-of-sight (NLOS) errors block the signal propagation between target device and base station. The NLOS errors have significant effects on ranging between two devices for wireless localization. In TOA-based localization, finding the target's location inside the overlapped area in the TOA-circles is difficult. We present an effective localization method using compensated distance with adaptive bias computation. The proposed method is possible for the target's location to estimate an accurate location in the overlapped area using the measured distances with subtracted adaptive bias. Through localization experiments in indoor environments, estimation error is reduced comparing to the conventional localization methods.

This paper addresses the problem of image alignment based on random measurements. Image alignment consists of estimating the relative transformation between a query image and a reference image. We consider the specific problem where the query image is provided in compressed form in terms of linear measurements captured by a vision sensor. We cast the alignment problem as a manifold distance minimization problem in the linear subspace defined by the measurements. The transformation manifold that represents synthesis of shift, rotation, and isotropic scaling of the reference image can be given in closed form when the reference pattern is sparsely represented over a parametric dictionary. We show that the objective function can then be decomposed as the difference of two convex functions (DC) in the particular case where the dictionary is built on Gaussian functions. Thus, the optimization problem becomes a DC program, which in turn can be solved globally by a cutting plane method. The quality of the solution is typically affected by the number of random measurements and the condition number of the manifold that describes the transformations of the reference image. We show that the curvature, which is closely related to the condition number, remains bounded in our image alignment problem, which means that the relative transformation between two images can be determined optimally in a reduced subspace.

Locating the center of the pupils is the most important foundation and the core component of gaze tracking. The accuracy of gaze tracking largely depends on the quality of images, but additional constraints and large amount of calculation make gaze tracking impractical on high-resolution images. Although some eye-gaze trackers can get accurate result, improving the accuracy of pupil feature on low-resolution images and accurately recognizing closed eye images are still common tasks in the field of gaze estimation. Our aim is to get the accurate localization of pupil center on low-resolution image. To this aim, we proposed a simple but effective method which can accurately locate pupil center in real time. The method first gets initial eye center based on improved scale-invariant feature transform (SIFT) descriptor and support vector machine (SVM) classifier, and then gets final position of the pupil center through a size variable correction rectangular block. In this paper, comparing with the reported state-of-the-art methods,the experimental results demonstrate that our system can achieve a more accurate result on low-resolution images. On top of that, our approach shows robustness on closed eye images while some other methods would not recognize the closed eye images.

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.32-2.43dB, 0.0036-0.0155, 0.0041-0.0198 and 0.0012-0.0057 respectively compared with competitive methods.

Any city-scale visual localization system has to overcome long-term appearance changes, such as varying illumination conditions or seasonal changes between query and database images. Since semantic content is more robust to such changes, we exploit semantic information to improve visual localization. In our scenario, the database consists of gnomonic views generated from panoramic images (e.g. Google Street View) and query images are collected with a standard field-of-view camera at a different time. To improve localization, we check the semantic similarity between query and database images, which is not trivial since the position and viewpoint of the cameras do not exactly match. To learn similarity, we propose training a CNN in a self-supervised fashion with contrastive learning on a dataset of semantically segmented images. With experiments we showed that this semantic similarity estimation approach works better than measuring the similarity at pixel-level. Finally, we used the semantic similarity scores to verify the retrievals obtained by a state-of-the-art visual localization method and observed that contrastive learning-based pose verification increases top-1 recall value to 0.90 which corresponds to a 2% improvement.

ABSTRACTDigital multimedia makes fabricating and copying much easier than ever before. Therefore, it demands efficient and automatic techniques to identify and verify the content of digital multimedia. Image authentication is such a technique to automatically identify whether the query image is a fabrication or a simple copy of the original one. In this paper, we propose a perceptual image authentication technique based on clustering and matching of feature points of images. Feature points are first extracted from images with the k‐largest local total variations and clustered using Fuzzy C‐means clustering algorithm. Then, feature points in the query image and the anchor image are matched into pairs in zigzag ordering along the diagonals of the images cluster by cluster. In the mean time, the outliers of feature points are removed. Then, the system decisions about the authenticity of images are determined by the majority vote of whether three types of distance between matched feature point pairs are larger than their respective thresholds. The three types of distance include the following: (i) histogram‐weighted distance, which is proposed in this paper; (ii) the normalized Euclidean distance; and (iii) the Hausdorff distance. The geometric transform between the query image and the anchor image is estimated, and the query image is registered. The possible tampered image blocks are detected, and the percentage of the tampered area is roughly estimated. The experimental results show the effectiveness and robustness of the proposed image authentication system. Copyright © 2011 John Wiley & Sons, Ltd.

Visual cross view geo-localization is generally approached within a joint retrieval-and-calibration framework. However, existing methods overlook semantic ambiguities arising from query and reference images characterized by low overlap, dynamic foregrounds, viewpoint changes, and perceptual aliasing. This makes it challenging to automatically control the relative importance of the two tasks, potentially compromising the retrieval task in favor of the offset regression. Consequently, the model may encounter conflicting dominating gradients during joint training. To address this, we propose to model the semantic ambiguity during the offset regression process by integrating associated uncertainty scores, represented as 2D Gaussian distributions, to mitigate negative transfer effects within the joint tasks. We further introduce an uncertainty-aware similarity metric to enhance similarity assessment between query and reference images, accounting for their semantic ambiguities. This metric propagates uncertainty scores into the retrieval task, focusing on certain samples and learning discriminative feature embeddings, allowing the model to adaptively handle conflicting dominating gradients during joint training. Extensive experiments demonstrate that our method improves the overall performance of the joint tasks, achieving state-of-the-art results on the VIGOR and CVACT datasets.

The objective of this paper is large scale object instance retrieval, given a query image. A starting point of such systems is feature detection and description, for example using SIFT. The focus of this paper, however, is towards very large scale retrieval where, due to storage requirements, very compact image descriptors are required and no information about the original SIFT descriptors can be accessed directly at run time. We start from VLAD, the state-of-the art compact descriptor introduced by Jegou et al. for this purpose, and make three novel contributions: first, we show that a simple change to the normalization method significantly improves retrieval performance, second, we show that vocabulary adaptation can substantially alleviate problems caused when images are added to the dataset after initial vocabulary learning. These two methods set a new state-of-the-art over all benchmarks investigated here for both mid-dimensional (20k-D to 30k-D) and small (128-D) descriptors. Our third contribution is a multiple spatial VLAD representation, MultiVLAD, that allows the retrieval and localization of objects that only extend over a small part of an image (again without requiring use of the original image SIFT descriptors).

Bag of Features (BoF) has been successfully exploited in content-based image retrieval for several years. Due to its performance and popularity, several extensions have been proposed that involve feature description, dictionary building, feature encoding and post-query process, etc. This paper proposes a dynamically weighting scheme for BoF-based image retrieval based on feedback. It involves two contributions: (i) analyzing the statistical distribution characteristic of similar BoF representations and (ii) computing weights dynamically based on the feedback obtained from different initial query results. We quantitatively evaluate the proposed method on two different databases. Experiments confirm that the proposed weighting scheme has better performance than the baseline of BoF-based image retrieval systems. Meanwhile, the results demonstrate the effectiveness of the weighting scheme in terms of the precision of top-N returned images.