Optimal Image Alignment With Random Projections of Manifolds: Algorithm and Geometric Analysis

Change detection, or anomaly detection, from street-view images acquired by an autonomous robot at multiple different times, is a major problem in robotic mapping and autonomous driving. Formulation as an image comparison task, which operates on a given pair of query and reference images is common to many existing approaches to this problem. Unfortunately, providing relevant reference images is not straightforward. In this paper, we propose a novel formulation for change detection, termed compressive change retrieval, which can operate on a query image and similar reference images retrieved from the web. Compared to previous formulations, there are two sources of difficulty. First, the retrieved reference images may frequently contain non-relevant reference images, because even state-of-the-art place-recognition techniques suffer from retrieval noise. Second, image comparison needs to be conducted in a compressed domain to minimize the storage cost of large collections of street-view images. To address the above issues, we also present a practical change detection algorithm that uses compressed bag-of-words (BoW) image representation as a scalable solution. The results of experiments conducted on a practical change detection task, "moving object detection (MOD)," using the publicly available Malaga dataset validate the effectiveness of the proposed approach.

This paper presents a new approach for the challenging problem of geo-localization using image matching in a structured database of city-wide reference images with known GPS coordinates. We cast the geo-localization as a clustering problem of local image features. Akin to existing approaches to the problem, our framework builds on low-level features which allow local matching between images. For each local feature in the query image, we find its approximate nearest neighbors in the reference set. Next, we cluster the features from reference images using Dominant Set clustering, which affords several advantages over existing approaches. First, it permits variable number of nodes in the cluster, which we use to dynamically select the number of nearest neighbors for each query feature based on its discrimination value. Second, this approach is several orders of magnitude faster than existing approaches. Thus, we obtain multiple clusters (different local maximizers) and obtain a robust final solution to the problem using multiple weak solutions through constrained Dominant Set clustering on global image features, where we enforce the constraint that the query image must be included in the cluster. This second level of clustering also bypasses heuristic approaches to voting and selecting the reference image that matches to the query. We evaluate the proposed framework on an existing dataset of 102k street view images as well as a new larger dataset of 300k images, and show that it outperforms the state-of-the-art by 20 and 7 percent, respectively, on the two datasets.

A new method for template-based image alignment is presented in this paper. A gradient-based optimization of motion compensated error difference is addressed to solve the image alignment problem. Its novelty lies in the minimized error, which considers the bi-directional compensation of the reference template and the current image onto each other in a constrained asymmetrical fashion. The proposed approach is shown to be a generalization of previous state-of-the art gradient-based methods. An experimental evaluation is provided to show how the new method outperforms the former in presence of noisy images, and to give some insights into its properties.

Motivated by the recent success of supervised and weakly supervised common object discovery, in this paper, we move forward one step further to tackle common object discovery in a fully unsupervised way. Generally, object co-localization aims at simultaneously localizing objects of the same class across a group of images. Traditional object localization/detection usually trains specific object detectors which require bounding box annotations of object instances, or at least image-level labels to indicate the presence/absence of objects in an image. Given a collection of images without any annotations, our proposed fully unsupervised method is to simultaneously discover images that contain common objects and also localize common objects in corresponding images. Without requiring to know the total number of common objects, we formulate this unsupervised object discovery as a sub-graph mining problem from a weighted graph of object proposals, where nodes correspond to object proposals, and edges represent the similarities between neighbouring proposals. The positive images and common objects are jointly discovered by finding sub-graphs of strongly connected nodes, with each sub-graph capturing one object pattern. The optimization problem can be efficiently solved by our proposed maximal-flow-based algorithm. Instead of assuming that each image contains only one common object, our proposed solution can better address wild images where each image may contain multiple common objects or even no common object. Moreover, our proposed method can be easily tailored to the task of image retrieval in which the nodes correspond to the similarity between query and reference images. Extensive experiments on PASCAL VOC 2007 and Object Discovery data sets demonstrate that even without any supervision, our approach can discover/localize common objects of various classes in the presence of scale, view point, appearance variation, and partial occlusions. We also conduct broad experiments on image retrieval benchmarks, Holidays and Oxford5k data sets, to show that our proposed method, which considers both the similarity between query and reference images and also similarities among reference images, can help to improve the retrieval results significantly.

In this paper we propose a panorama-based annotation scheme for superimposing annotation information on broadcast video images by using a set of panoramic images, annotation information for each image and the adjacency relations between these images as information sources. The proposed method aligns an input video sequence with a set of panoramic reference images and displays annotation information on those parts of the video image that correspond to locations annotated in these panoramic images. The method first aligns locations on each of the panoramic images with those in the input video sequence and then selects that panoramic image that was taken from the position that is closest to that of the input image and uses this as a reference for annotation. The method uses mean squared error (MSE) of pixel intensities between the two images as a criterion for alignment. Annotation information is then displayed based on the alignment results with the selected panoramic image. In order to display annotation information appropriately when the camera viewpoint moves, we perform alignment of the input image with the panoramic images that are adjacent to the reference image in parallel, and switch over to another panoramic image taking it as the new reference image if this gives better alignment results than the current reference image. By monitoring the progress of such changeovers between images we can estimate the trajectory of the camera and its position. We have implemented the proposed method on a computer system and obtained favorable results regarding its effectiveness and the suitability of the method to real-time processing. © 2006 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 89(11): 51–60, 2006; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20282

Finding an image's exact GPS location is a challenging computer vision problem that has many real-world applications. In this paper, we address the problem of finding the GPS location of images with an accuracy which is comparable to hand-held GPS devices. We leverage a structured data set of about 100,000 images build from Google Maps Street View as the reference images. We propose a localization method in which the SIFT descriptors of the detected SIFT interest points in the reference images are indexed using a tree. In order to localize a query image, the tree is queried using the detected SIFT descriptors in the query image. A novel GPS-tag-based pruning method removes the less reliable descriptors. Then, a smoothing step with an associated voting scheme is utilized; this allows each query descriptor to vote for the location its nearest neighbor belongs to, in order to accurately localize the query image. A parameter called Confidence of Localization which is based on the Kurtosis of the distribution of votes is defined to determine how reliable the localization of a particular image is. In addition, we propose a novel approach to localize groups of images accurately in a hierarchical manner. First, each image is localized individually; then, the rest of the images in the group are matched against images in the neighboring area of the found first match. The final location is determined based on the Confidence of Localization parameter. The proposed image group localization method can deal with very unclear queries which are not capable of being geolocated individually.

In this paper we use the cumulative distribution of a random variable to define the information content in it and use it to develop a novel measure of information that parallels Shannon entropy, which we dub cumulative residual entropy (CRE). The key features of CRE may be summarized as, (1) its definition is valid in both the continuous and discrete domains, (2) it is mathematically more general than the Shannon entropy and (3) its computation from sample data is easy and these computations converge asymptotically to the true values. We define the cross-CRE (CCRE) between two random variables and apply it to solve the uni- & multimodal image alignment problem for parameterized (rigid, affine and projective) transformations. The key strengths of the CCRE over using the now popular mutual information method (based on Shannon’s entropy) are that the former has significantly larger noise immunity and a much larger convergence range over the field of parameterized transformations. These strengths of CCRE are demonstrated via experiments on synthesized and real image data. 1.

Visual cross view geo-localization is generally approached within a joint retrieval-and-calibration framework. However, existing methods overlook semantic ambiguities arising from query and reference images characterized by low overlap, dynamic foregrounds, viewpoint changes, and perceptual aliasing. This makes it challenging to automatically control the relative importance of the two tasks, potentially compromising the retrieval task in favor of the offset regression. Consequently, the model may encounter conflicting dominating gradients during joint training. To address this, we propose to model the semantic ambiguity during the offset regression process by integrating associated uncertainty scores, represented as 2D Gaussian distributions, to mitigate negative transfer effects within the joint tasks. We further introduce an uncertainty-aware similarity metric to enhance similarity assessment between query and reference images, accounting for their semantic ambiguities. This metric propagates uncertainty scores into the retrieval task, focusing on certain samples and learning discriminative feature embeddings, allowing the model to adaptively handle conflicting dominating gradients during joint training. Extensive experiments demonstrate that our method improves the overall performance of the joint tasks, achieving state-of-the-art results on the VIGOR and CVACT datasets.

The digitization of ancient murals is crucial in preserving, inheriting, and utilizing cultural heritage. Due to the vast coverage of murals, digitization typically involves capturing images in segments and then stitching them together. However, existing image stitching techniques face limitations regarding efficiency, generalization capabilities, and robustness, which hinder their practical applicability. To solve these problems and improve the performance of image stitching, this paper proposes an unsupervised hierarchical grid-constrained fusion network model. This model consists of two main modules: image alignment and image synthesis. The image alignment module incorporates prior knowledge, such as feature pyramids, attention mechanisms, and context dependencies, to facilitate feature extraction. It also includes a multi-scale grid homography generation part that utilizes region masks to create a deformation field. Additionally, a stitching-domain transformation unit is employed to ensure spatial consistency by deforming the input reference and target images. In the image synthesis module, a progressive inference fusion network framework is proposed to simplify the complex image fusion problem into a multi-granularity image synthesis problem. This framework utilizes two encoder-decoder cascaded network units to merge information progressively from coarse to fine granularity, producing high-resolution stitched images. Experimental results demonstrate that the proposed model exhibits superior robustness on both public and custom image datasets, and generally outperforms state-of-the-art image stitching methods, especially in subjective visual assessments.

Identifying the manufacturer and the type of cardiac implantable electronic devices (CIEDs) is important in emergent clinical settings. Recent studies have illustrated that artificial neural network models can successfully recognize CIEDs from chest X-ray images. However, all existing methods require a vast amount of medical data to train the classification model. Here, we have proposed a novel method to retrieve an identical CIED image from an image database by employing the feature point matching algorithm. A total of 653 unique X-ray images from 456 patients who visited our pacemaker clinic between April 2012 and August 2020 were collected. The device images were manually square-shaped, and was thereafter resized to 224 × 224 pixels. A scale-invariant feature transform (SIFT) algorithm was used to extract the keypoints from the query image and reference images. Paired feature points were selected via brute-force matching, and the average Euclidean distance was calculated. The image with the shortest average distance was defined as the most similar image. The classification performance was indicated by accuracy, precision, recall, and F1-score for detecting the manufacturers and model groups, respectively. The average accuracy, precision, recall, and F-1 score for the manufacturer classification were 97.0%, 0.97, 0.96, and 0.96, respectively. For the model classification task, the average accuracy, precision, recall, and F-1 score were 93.2%, 0.94, 0.92, and 0.93, respectively, all of which were higher than those of the previously reported machine learning models. Feature point matching is useful for identifying CIEDs from X-ray images.

Abstract. In this paper, we present our approach for robust long-term visual localization in large scale urban environments exploiting street level imagery. Our approach consists of a 2D-image based localization using image retrieval (NetVLAD) to select reference images. This is followed by a 3D-structure based localization with a robust image matcher (DenseSfM) for accurate pose estimation. This visual localization approach is evaluated by means of the ‘Sun’ subset of the RobotCar seasons dataset, which is part of the Visual Localization benchmark. As the results on the RobotCar benchmark dataset are nearly on par with the top ranked approaches, we focused our investigations on reproducibility and performance with own data. For this purpose, we created a dataset with street-level imagery. In order to have independent reference and query images, we used a road-based and a tram-based mapping campaign with a time difference of four years. The approximately 90% successfully oriented images of both datasets are a good indicator for the robustness of our approach. With about 50% success rate, every second image could be localized with a position accuracy better than 0.25 m and a rotation accuracy better than 2°.

To quickly synthesize complex scenes, digital artists often collage together visual elements from multiple sources: for example, mountains from New Zealand behind a Scottish castle with wisps of Saharan sand in front. In this paper, we propose to use a similar process in order to parse a scene. We model a scene as a collage of warped, layered objects sampled from labeled, reference images. Each object is related to the rest by a set of support constraints. Scene parsing is achieved through analysis-by-synthesis. Starting with a dataset of labeled exemplar scenes, we retrieve a dictionary of candidate object segments that match a query image. We then combine elements of this set into a "scene collage" that explains the query image. Beyond just assigning object labels to pixels, scene collaging produces a lot more information such as the number of each type of object in the scene, how they support one another, the ordinal depth of each object, and, to some degree, occluded content. We exploit this representation for several applications: image editing, random scene synthesis, and image-to-anaglyph.

Visual Place Recognition (VPR) is process of properly identifying a formerly visited place under varying viewpoints conditions. VPR becomes a challenging task due to the variations in lighting conditions (day time or night time), shadows, weather conditions, view points, or even different seasons. Therefore, it is needed to develop VPR algorithms which can handle the variation in visual appearances. This article designs a visual place recognition using a novel region of interest (RoI) extraction with deep learning based (VPR-ROIDL) technique on changing environments. The proposed VPR-ROIDL technique initially enables the extraction of RoIs using saliency map then features of RoIs using local diagonal extrema patterns (LDEP) respectively. Besides, the extracted features of RoIs are passed into the convolutional neural network based residual network (ResNet) model and the computed deep features are stored in the database. Upon providing a query image (QI), the RoIs and features of RoIs are extracted as same as reference images. Moreover, the deep features from the ResNet model are generated and Euclidean distance based similarity measurement is used to finally recognize the places. By the use of RoI and descriptors matching process, the VPR-ROIDL model has the ability to recognize places irrespective of changing illuminations, seasons, and viewpoints. The simulation results pointed out the supremacy of the ROIDL-VPR technique over the recent state of art approaches.KeywordsVisual Place recognitionDeep learningVisual appearancesRegion of InterestSaliency mapSimilarity Measurement

Ear images have been shown to be a reliable modality for biometric recognition with desirable characteristics, such as high universality, distinctiveness, measurability and permanence. While a considerable amount of research has been directed towards ear recognition techniques, the problem of ear alignment is still under‐explored in the open literature. Nonetheless, accurate alignment of ear images, especially in unconstrained acquisition scenarios, where the ear appearance is expected to vary widely due to pose and view point variations, is critical for the performance of all downstream tasks, including ear recognition. Here, the authors address this problem and present a framework for ear alignment that relies on a two‐step procedure: (i) automatic landmark detection and (ii) fiducial point alignment. For the first (landmark detection) step, the authors implement and train a Two‐Stack Hourglass model (2‐SHGNet) capable of accurately predicting 55 landmarks on diverse ear images captured in uncontrolled conditions. For the second (alignment) step, the authors use the Random Sample Consensus (RANSAC) algorithm to align the estimated landmark/fiducial points with a pre‐defined ear shape (i.e. a collection of average ear landmark positions). The authors evaluate the proposed framework in comprehensive experiments on the AWEx and ITWE datasets and show that the 2‐SHGNet model leads to more accurate landmark predictions than competing state‐of‐the‐art models from the literature. Furthermore, the authors also demonstrate that the alignment step significantly improves recognition accuracy with ear images from unconstrained environments compared to unaligned imagery.

Optical target recognition using correlators is an important technique for fast verification and identification of images. The hybrid opto-electronic correlator (HOC) recently proposed by us bypasses the need for nonlinear materials such as photorefractive polymer films by using detectors instead, and the phase information is yet conserved by the interference of plane waves with the images. In this paper, we demonstrate experimentally the basic working principle of the HOC architecture using currently available technologies. For matched reference and query images, the output signal shows a sharp peak, indicating a match is found. For an unmatched case, a much lower peak value is observed, indicating no match. We also demonstrate the dependence of the output signal on the phases of the interfering plane waves and describe a technique using an interferometer and a servo for optimizing the output signal. As such, the work reported here paves the way for further development of the HOC for practical applications.