Neighborhood matching for object recognition algorithms based on local image features

Conv-Swinformer: Integration of CNN and shift window attention for Alzheimer’s disease classification

AEA-Net:Affinity-supervised entanglement attentive network for person re-identification

The vast growth of image databases creates many challenges for computer vision applications, for instance image retrieval and object recognition. Large variation in imaging conditions such as illumination and geometrical properties (including scale, rotation, and viewpoint) gives rise to the need for invariant features; i.e. image features should have minimal differences under these conditions. Local image features in the form of key points are widely used because of their invariant properties. In this chapter, we analyze different issues relating to existing local feature detectors. Based on this analysis, we present a new approach for detecting and filtering local features. The proposed approach is tested in a real-life application which supports navigation in urban environments based on visual information. The study shows that our approach performs as well as existing methods but with a significantly lower number of features.KeywordsFeature VectorLocal FeatureInterest PointQuery ImageImage RecognitionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In person re-identification, extracting image features is an important step when retrieving pedestrian images. Most of the current methods only extract global features or local features of pedestrian images. Some inconspicuous details are easily ignored when learning image features, which is not efficient or robust to for scenarios with large differences. In this paper, we propose a Multi-level Feature Fusion model that combines both global features and local features of images through deep learning networks to generate more discriminative pedestrian descriptors. Specifically, we extract local features from different depths of network by the Part-based Multi-level Net to fuse low-to-high level local features of pedestrian images. Global-Local Branches are used to extract the local features and global features at the highest level. The experiments have proved that our deep learning model based on multi-level feature fusion works well in person re-identification. The overall results outperform the state of the art with considerable margins on three widely-used datasets. For instance, we achieve 96% Rank-1 accuracy on the Market-1501 dataset and 76.1% mAP on the DukeMTMC-reID dataset, outperforming the existing works by a large margin (more than 6%).

Person re-identification aims to match person of interest under non-overlapping camera views. Therefore, how to generate a robust and discriminative representation is crucial for person re-identification. Mining local clues from human body parts to describe pedestrians has been extensively studied in existing methods. However, existing methods locate human body parts coarsely and do not consider the relations among different local parts. To address the above problem, we propose a Part-based Structured Representation Learning (PSRL) for better exploiting local clues to improve the person representation. There are two important modules in our architecture: Local Semantic Feature Extraction and Structured Person Representation Learning. The Local Semantic Feature Extraction module is designed to extract local features from human body semantic regions. After obtaining the local features, the Structured Person Representation Learning is proposed to fuse the local features by considering the person structure. To model the underlying person structure, a graph convolutional network is employed to capture the relations of different semantic regions. The generated structured feature encodes underlying person structure information, and local semantic feature can solve the misalignment problem caused by pose variations in feature matching. By combining them together, we can improve the descriptive ability of the generated representation. Extensive evaluations on four standard benchmarks show that our proposed method achieves competitive performance against state-of-the-art methods.

SADGFeat: Learning local features with layer spatial attention and domain generalization

Surveillance cameras can be deployed in various environments where lighting conditions are constantly changing. However, due to the limited dynamic range of current image sensors, the captured images are only low dynamic range images that usually suffer from over-exposure and under-exposure situations where important details are lost. Therefore, it is critical to recover the lost details of such images in order to improve visual experience for observers and performance for possible computer vision processing. In this paper, we propose a reformulated Laplacian pyramid and a convolutional neural network (CNN) model to enhance and recover the lost detail of a degraded image. The reformulated Laplacian first decomposes the image into two sub-images that contain global and local image features, respectively. The global features and local features are processed by the proposed CNN model to manipulate the global luminance terrain and enhance local details. The final image is obtained by reconstructing the CNN generated local and global features. Various experiments have been conducted. The results demonstrate that the proposed model outperforms the state-of-the-art methods.

A new indexing system for multiresolution image retrieval has been developed. The system analyzes not only whole images but also--for the first time--local features present in the images. This capability is important in managing large-scale image systems such as video databases and medical image databases. Key to the method is the use of an innovative wavelet, called the W-transform. This transform is characterized by its ability to handle signals of arbitrary size. Equally important, the W-transform is not constrained by assumptions about periodicity. That is, the transformed signal in some location is related only to the signals at its neighboring location. These properties make the W-transform a natural tool for analyzing local image features. A brief discussion is given of how the system works. The feature-oriented image indexing and retrieval method offers two major advantages over traditional multiresolution histogram comparison methods. The response time for retrieving image coarse components is considerably faster than that for retrieving whole images--thereby dramatically speeding image retrieval over the Ethernet. Moreover, the new method can be used to search images that are significantly different from the query image as a whole but contain local features identical with or similar to those in the query image.

Matching local features across images is often useful when comparing or recognizing objects or scenes, and efficient techniques for obtaining image-to-image correspondences have been developed [4, 11, 16]. However, given a query image, searching a very large image database with such measures remains impractical. We introduce a sub-linear time randomized hashing algorithm for indexing sets of feature vectors under their partial correspondences. We develop an efficient embedding function for the normalized partial matching similarity between sets, and show how to exploit random hyperplane properties to construct hash functions that satisfy locality-sensitive constraints. The result is a bounded approximate similarity search algorithm that finds (1 + epsiv)-approximate nearest neighbor images in O(N1/1+epsiv) time for a database containing N images represented by (varying numbers of) local features. We demonstrate our approach applied to image retrieval for images represented by sets of local appearance features, and show that searching over correspondences is now scalable to large image databases.

During the continual transitions from lab research to real-world applications of vision-based algorithms, there are significant challenges, e.g. the robustness to adapt to complex environments and the high demands of multi-task, multi-modal learning models. More specifically on visual simultaneous localization and mapping (visual SLAM) for mobile robots, two notable limitations are: (i) the drift issue during pose estimation, especially in dynamic environments, makes the positioning system unstable; (ii) the additional loop detection takes many computational resources for image retrieval and global feature geometry checking. This thesis explores the approaches of visual localization on unmanned ground vehicles by solving those limitations. The conventional feature extraction and matching pipeline for vision-based tasks involve three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. The first contribution of the thesis is adding a feature selection and anticipation stage to reduce tracking drift. We explore the noise model of image features to select a subset of all the observed image features with the best ”contribution” during data association and pose estimation across multiple frames. Conventional SLAM algorithms take a strong assumption of scene rigidity, which limits the application under challenging environments. The second part of the thesis addresses the tough issue in dynamic environments with moving objects. We presented GMC, namely the motion clustering approach, a lightweight dynamic object filtering method. It can distinguish moving objects from static landmarks. Based on the theory of motion coherence within a particular image area, GMC could segment dynamic objects in 3D space. We can provide an efficient and robust correspondence algorithm that can extract dynamic objects from a static background with the method. In this way, we propose a dynamic SLAM system that is real-time and free from expensive GPU processors. In contrast to GMC, the thesis’s third part turns to an object-aware learning-based model for more general dynamic scenarios. We use object detection and tracking as points, lines, planes, etc. We utilize semantic information and extract sparse image features simultaneously to keep track of dynamic objects. The static background and different dynamic objects are jointly optimized in a newly developed bundle adjustment sliding window. The estimated 3D bounding boxes can provide more robust camera tracking and better scene understanding, and better map merging. The fourth part of the thesis leverages the emerging feature learning framework. It proposes a unified self-supervised model called LGDNet to generate both global and local image feature descriptors end-to-end. Global feature descriptors embed the whole image into a compact representation, leading to easier scene comparison. On the other hand, local features focus more on the local region similarities of some salient parts for structure from motion. Our proposed method can directly extract features together with descriptors that encode both local maximum responses and global context information, avoiding duplicate calculations based on different feature extraction criteria.

The problem of detecting local image features that are invariant to scale, orientation, illumination and viewpoint changes is a critical issue in many computer vision applications. The challenges involve localizing the image features accurately in the spatial and frequency domains and describing them with a stable analytical representation. In this paper we address these two issues by proposing a new non-linear scale-space implementation that improves the localization accuracy of the SIFT [3] local features. Furthermore we propose a simple adjustment to the standard SIFT descriptor and show that the modified version is more robust to affine changes.

Addressing the limitations in current visual question answering (VQA) models face limitations in multimodal feature fusion capabilities and often lack adequate consideration of local information, this study proposes a multimodal Transformer VQA network based on local and global information integration (LGMTNet). LGMTNet employs attention on local features within the context of global features, enabling it to capture both broad and detailed image information simultaneously, constructing a deep encoder-decoder module that directs image feature attention based on the question context, thereby enhancing visual-language feature fusion. A multimodal representation module is then designed to focus on essential question terms, reducing linguistic noise and extracting multimodal features. Finally, a feature aggregation module concatenates multimodal and question features to deepen question comprehension. Experimental results demonstrate that LGMTNet effectively focuses on local image features, integrates multimodal knowledge, and enhances feature fusion capabilities.

This chapter describes a novel appearance-based approach for topological mapping called HTMap (Hierarchical Topological Mapping), which is based on a hierarchical decomposition of the environment. Images with similar appearances are grouped together in locations, taking as a representative of the group the average of the PHOG global descriptors of the represented images, as well as the set of their local features, which are indexed by means of OBIndex (which handles them as explained in the previous chapter). As a main innovation, the algorithm proposes a two-level approach to detect loop candidates: first, the global descriptor of the current image is used to determine the most similar location of the map; next, local image features are employed to determine the most likely image within that location.

We introduce Object-Guided localization (OGuL) based on a novel method of local-feature matching. Direct matching of local features is sensitive to significant changes in illumination. In contrast, object detection often survives severe changes in lighting conditions. The proposed method first detects semantic objects and establishes correspondences of those objects between images. Object correspondences provide local coarse alignment of the images in the form of a planar homography. These homographies are consequently used to guide the matching of local features. Experiments on standard urban localization datasets (Aachen, RobotCar-Season) show that OGuL significantly improves localization results with as simple local features as SIFT, and its performance competes with the state-of-the-art CNN-based methods trained for day-to-night localization.

Local image features matching is a useful technique for 3D reconstruction and localization in ITS field. In recent years, machine learning-based local feature matching methods have been proposed. However, it is impractical to manually generate the ground truth of corresponding points used as training data on these methods. Therefore, for in-vehicle camera images we propose an automatic training data collection method using geometric constraints and continuity with the movement of the vehicle. The experimental results show that the proposed method can provide effective corresponding points for training a machine learning-based matcher.