Binary Set Embedding for Cross-Modal Retrieval

Hashing has been extensively utilized in cross-modal retrieval due to its high efficiency in handling large-scale, high-dimensional data. However, most existing cross-modal hashing methods operate as offline learning models, which learn hash codes in a batch-based manner and prove to be inefficient for streaming data. Recently, several online cross-modal hashing methods have been proposed to address the streaming data scenario. Nevertheless, these methods fail to fully leverage the semantic information and accurately optimize hashing in a discrete fashion. As a result, both the accuracy and efficiency of online cross-modal hashing methods are not ideal. To address these issues, this paper introduces the Semantic Embedding-based Online Cross-modal Hashing (SEOCH) method, which integrates semantic information exploitation and online learning into a unified framework. To exploit the semantic information, we map the semantic labels to a latent semantic space and construct a semantic similarity matrix to preserve the similarity between new data and existing data in the Hamming space. Moreover, we employ a discrete optimization strategy to enhance the efficiency of cross-modal retrieval for online hashing. Through extensive experiments on two publicly available multi-label datasets, we demonstrate the superiority of the SEOCH method.

The image-recipe cross-modal retrieval task, which retrieves the relevant recipes according to food images and vice versa, is now attracting widespread attention. There are two main challenges for image-recipe cross-modal retrieval task. Firstly, a recipe’s different components (words in a sentence, sentences in an entity, and entities in a recipe) have different weight values. If a recipe’s different components own the same weight, the recipe embeddings cannot pay more attention to the important components. As a result, the important components make less contribution to the retrieval task. Secondly, the food images have obvious properties of locality and only the local food regions matter. There are still difficulties in enhancing the discriminative local region features in the food images. To address these two problems, we propose a novel framework named Dual Cross Attention Encoders for Cross-modal Food Retrieval (DCA-Food). The proposed framework consists of a hierarchical cross attention recipe encoder (HCARE) and a cross attention image encoder (CAIE). HCARE consists of three types of cross attention modules to capture the important words in a sentence, the important sentences in an entity and the important entities in a recipe, respectively. CAIE extracts global and local region features. Then, it calculates cross attention between them to enhance the discriminative local features in the food images. We conduct the ablation studies to validate our design choices. Our proposed approach outperforms the existing approaches by a large margin on the Recipe1M dataset. Specifically, we improve the R@1 performance by +2.7 and +1.9 on the 1k and 10k testing sets, respectively.

Multi-label enhancement based self-supervised deep cross-modal hashing

Conv-Swinformer: Integration of CNN and shift window attention for Alzheimer’s disease classification

Discriminative correlation hashing for supervised cross-modal retrieval

In person re-identification, extracting image features is an important step when retrieving pedestrian images. Most of the current methods only extract global features or local features of pedestrian images. Some inconspicuous details are easily ignored when learning image features, which is not efficient or robust to for scenarios with large differences. In this paper, we propose a Multi-level Feature Fusion model that combines both global features and local features of images through deep learning networks to generate more discriminative pedestrian descriptors. Specifically, we extract local features from different depths of network by the Part-based Multi-level Net to fuse low-to-high level local features of pedestrian images. Global-Local Branches are used to extract the local features and global features at the highest level. The experiments have proved that our deep learning model based on multi-level feature fusion works well in person re-identification. The overall results outperform the state of the art with considerable margins on three widely-used datasets. For instance, we achieve 96% Rank-1 accuracy on the Market-1501 dataset and 76.1% mAP on the DukeMTMC-reID dataset, outperforming the existing works by a large margin (more than 6%).

AEA-Net:Affinity-supervised entanglement attentive network for person re-identification

In recent years, increasing amounts of video resources have created a series of demands for fine retrieval of video moments, such as highlight moments in sports events and the re-creation of specific video content. In this context, research on cross-modal video segment retrieval, which attempts to output a video moment that matches the input query text, is gradually emerging. Existing solutions primarily focus on global or local feature representation for query text and video moments. However, such solutions ignore matching semantic relations contained in query text and video moments. For example, given the query text “a person is playing basketball, existing retrieval systems may incorrectly return a video moment of “a person holding a basketball without the considering the semantic relationship of “a person playing basketball. Therefore, this paper proposes a cross-modal relationship alignment framework, which we refer to as CrossGraphAlign, for cross-modal video moment retrieval. The proposed framework constructs a textual relationship graph and a visual relationship graph to model the query semantics in text and video segment relations, and then evaluates the similarity between text relations and visual relations through cross-modally aligned graph convolutional networks to help construct a more accurate video moment retrieval system. Experimental results on the publicly available cross-modal video retrieval datasets TACoS and ActivityNet Captions demonstrate that the proposed method can effectively utilize the semantic relationships to improve the recall rate in cross-modal video moment retrieval.

Perceptual encryption has received widespread attention as a technology for protecting multimedia visual information. In multimedia applications, perceptual encryption only protects a portion of the content without previewing all the information. At present, perceptual encryption is mostly oriented to conventional plain images while few methods aim at visual security measures for perceptually encrypted images. Existing solutions usually adopt well-known quality assessment metrics to measure the visual quality of encrypted images. However, they often exhibit undesired behavior on perceptually encrypted images with low quality. As a typical representation of three-dimensional scenes, light field images record the intensity and direction of light during propagation which is distinct from 2-D images. In this paper, we construct a perceptually encrypted light field image database (PE-LFID) for quality assessment based on 14 reference light field plaintext images. For each scene, we employ four encryption methods, each of which has six levels. Additionally, a novel visual security evaluation method based on PE-LFID is proposed by taking into account the local and global features of the light field images. First, we use a multi-threshold edge detection method to obtain the edge similarity in the spatial domain of the light field image. Afterwards, the epipolar plane image (EPI) generated from the angular domain of the light field image is used to calculate the gradient magnitude similarity, which is expressed as a global feature. Furthermore, the final quality prediction score is calculated by adaptively weighting between local and global features. We conduct extensive experiments on the proposed PE-LFID to assess the performance of classical and state-of-the-art IQA models. The experimental results demonstrate the effectiveness of the proposed method for visual security evaluation of perceptually encrypted light field images, as well as the scalability of PE-LFID.

Different domain image sensors or imaging mechanisms provide cross-domain images when sensing the same scene. There is a domain shift between cross-domain images so that the image gap between different domains is the major challenge for measuring the similarity of the feature descriptors extracted from different domain images. Specifically, matching ground camera images and unmanned aerial vehicle (UAV) 3-D model-rendered images, which are two kinds of extremely challenging cross-domain images, is a way to establish indirectly the spatial relationship between 2-D and 3-D spaces. This provides a solution for the virtual-real registration of augmented reality (AR) in outdoor environments. However, during matching, handcrafted descriptors and existing learning-based feature descriptors limit the rendered images. In this letter, first, to learn robust and invariant 128-D local feature descriptors for ground camera and rendered images, we present a novel network structure, SiamAM-Net, which embeds the autoencoders with an attention mechanism into the Siamese network. Then, to narrow the gap between the cross-domain images during the optimizing of SiamAM-Net, we design an adaptive margin for the loss function. Finally, we match the ground camera-rendered images by using the learned local feature descriptors and explore the outdoor AR virtual-real registration. Experiments show that the local feature descriptors, learned by SiamAM-Net, are robust and achieve state-of-the-art retrieval performance on the cross-domain image data set of ground camera and rendered images. In addition, several outdoor AR applications also demonstrate the usefulness of the proposed outdoor AR virtual-real registration.

This paper proposes a hybrid method for face recognition using local features and statistical feature extraction methods. First, a dense set of local feature points are extracted in order to represent a facial image. Each local feature point is described by the keypoint descriptor defined by SIFT feature. Then, the statistical feature extraction methods, PCA and LDA, are applied to the set of local feature descriptors in order to find low dimensional features. With the obtained low dimensional feature vectors, we can conduct face recognition task efficiently using a simple classifier. Through computational experiments on benchmark data sets, we show that the proposed method is superior to the conventional PCA and LDA in the classification performance. In addition, we also show that the proposed method can achieve remarkable improvement in the processing time compared to the conventional keypoint matching methods proposed for local features.KeywordsFace RecognitionLocal FeatureSift FeatureGray Level IntensityConventional Statistical MethodThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The vast growth of image databases creates many challenges for computer vision applications, for instance image retrieval and object recognition. Large variation in imaging conditions such as illumination and geometrical properties (including scale, rotation, and viewpoint) gives rise to the need for invariant features; i.e. image features should have minimal differences under these conditions. Local image features in the form of key points are widely used because of their invariant properties. In this chapter, we analyze different issues relating to existing local feature detectors. Based on this analysis, we present a new approach for detecting and filtering local features. The proposed approach is tested in a real-life application which supports navigation in urban environments based on visual information. The study shows that our approach performs as well as existing methods but with a significantly lower number of features.KeywordsFeature VectorLocal FeatureInterest PointQuery ImageImage RecognitionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Surveillance cameras can be deployed in various environments where lighting conditions are constantly changing. However, due to the limited dynamic range of current image sensors, the captured images are only low dynamic range images that usually suffer from over-exposure and under-exposure situations where important details are lost. Therefore, it is critical to recover the lost details of such images in order to improve visual experience for observers and performance for possible computer vision processing. In this paper, we propose a reformulated Laplacian pyramid and a convolutional neural network (CNN) model to enhance and recover the lost detail of a degraded image. The reformulated Laplacian first decomposes the image into two sub-images that contain global and local image features, respectively. The global features and local features are processed by the proposed CNN model to manipulate the global luminance terrain and enhance local details. The final image is obtained by reconstructing the CNN generated local and global features. Various experiments have been conducted. The results demonstrate that the proposed model outperforms the state-of-the-art methods.

Local image features around interest-points have been widely used in order to exploit the similarities between different views of an object in different images. While there are numerous algorithms on detecting the interest-points and defining the local features, few have focused on the importance of the matching process. In this paper, we presented a method that matches interest-points detected via any algorithm. The method is motivated from human perceptual rules, particularly the Gestalt Psychology, and realizes the fact that “The whole is different from the sum of its parts”. The efficacy of the algorithm is not only the ability to decrease the number of false positive matches but also to increase the number of true positives, yielding rock-steady results for any algorithm based on matching local features.

A new indexing system for multiresolution image retrieval has been developed. The system analyzes not only whole images but also--for the first time--local features present in the images. This capability is important in managing large-scale image systems such as video databases and medical image databases. Key to the method is the use of an innovative wavelet, called the W-transform. This transform is characterized by its ability to handle signals of arbitrary size. Equally important, the W-transform is not constrained by assumptions about periodicity. That is, the transformed signal in some location is related only to the signals at its neighboring location. These properties make the W-transform a natural tool for analyzing local image features. A brief discussion is given of how the system works. The feature-oriented image indexing and retrieval method offers two major advantages over traditional multiresolution histogram comparison methods. The response time for retrieving image coarse components is considerably faster than that for retrieving whole images--thereby dramatically speeding image retrieval over the Ethernet. Moreover, the new method can be used to search images that are significantly different from the query image as a whole but contain local features identical with or similar to those in the query image.