Deep Supervised Hashing for Fast Image Retrieval

Purpose To evaluate the robustness of an award-winning bone age deep learning (DL) model to extensive variations in image appearance. Materials and Methods In December 2021, the DL bone age model that won the 2017 RSNA Pediatric Bone Age Challenge was retrospectively evaluated using the RSNA validation set (1425 pediatric hand radiographs; internal test set in this study) and the Digital Hand Atlas (DHA) (1202 pediatric hand radiographs; external test set). Each test image underwent seven types of transformations (rotations, flips, brightness, contrast, inversion, laterality marker, and resolution) to represent a range of image appearances, many of which simulate real-world variations. Computational "stress tests" were performed by comparing the model's predictions on baseline and transformed images. Mean absolute differences (MADs) of predicted bone ages compared with radiologist-determined ground truth on baseline versus transformed images were compared using Wilcoxon signed rank tests. The proportion of clinically significant errors (CSEs) was compared using McNemar tests. Results There was no evidence of a difference in MAD of the model on the two baseline test sets (RSNA = 6.8 months, DHA = 6.9 months; P = .05), indicating good model generalization to external data. Except for the RSNA dataset images with an appended radiologic laterality marker (P = .86), there were significant differences in MAD for both the DHA and RSNA datasets among other transformation groups (rotations, flips, brightness, contrast, inversion, and resolution). There were significant differences in proportion of CSEs for 57% of the image transformations (19 of 33) performed on the DHA dataset. Conclusion Although an award-winning pediatric bone age DL model generalized well to curated external images, it had inconsistent predictions on images that had undergone simple transformations reflective of several real-world variations in image appearance. Keywords: Pediatrics, Hand, Convolutional Neural Network, Radiography Supplemental material is available for this article. © RSNA, 2024 See also commentary by Faghani and Erickson in this issue.

Deep hashing methods utilize an end-to-end framework to mutually learn feature representations and hash codes, thereby achieving a better retrieval performance. Traditional supervised hashing methods adopt handcrafted features for hashing function learning and then generate hash codes through classification and quantization. The lack of adaptability and independence of the quantization procedure leads to low retrieval accuracy of supervised hashing methods with handcrafted features in image retrieval. In this study, a non-relaxation deep hashing method for fast image retrieval is proposed. In this method, a differentiable host thresholding function is used to encourage hash-like codes to approach -1 or 1 non-linearly at the output of the convolutional neural, instead of the symbol function for quantization used in the traditional method. The output of the host thresholding function is directly used to compute the network training error, and a loss function is elaborately designed with the norm to constrain each bit of the hash-like code to be as binary as possible. Finally, a symbol function is added outside the trained network model to generate binary hash codes for image storage and retrieval in a low-dimensional binary space. Extensive experiments on two large-scale public datasets show that our method can effectively learn image features, generate accurate binary hash codes, and outperform state-of-the-art methods in terms of the mean average precision.

Hashing methods have been widely used in large-scale image retrieval. However, the constraints on the hash codes of similar images learned by the previous hashing methods are too strong, which may lead to overfitting and difficult convergence. Besides, the binary codes output by the previous hashing methods are not optimally compatible with the multi-index approach, which is the most effective method for Hamming distance query acceleration. In this paper, we propose a novel Robust and Index-Compatible Deep Hashing (RICH) method to learn compact similarity-preserving binary codes, which focuses on improving the retrieval accuracy and time efficiency simultaneously. With the learned binary codes, we can achieve better results compared with the state-of-the-arts in retrieval accuracy. Meanwhile, remarkable promotions of the retrieval time efficiency have been made in the Hamming distance query process.

Object-oriented convolutional features for fine-grained image retrieval in large surveillance datasets

Visual search in millions of samples in high-dimensional feature space is computationally expensive and challenging. A natural solution is to reduce the dimension of image representation by mapping each sample into compact binary code. In this paper, we propose a Ranking Based Semantic Hashing (RBSH) method to tackle with this problem. Observing that semantic structures carry complementary information, this paper takes advantage of semantic supervision for training high quality hashing, the semantic mapping between the high-dimensional feature space of samples and the reduced representation space of binary code, with the help of pre-trained word2vec. Specifically, the proposed method learns the mapping based on two criteria: the contrastive ranking loss and the orthogonality constraint. The former preserves the ordering of relative similarity in image pairs, while the latter makes different bit in the hash stream as orthogonal as possible. Extensive experimental study has been conducted on VOC2012 and ILSVRC2014 image sets, demonstrating that the proposed approach generally outperforms the state-of-the-art hashing techniques based methods in image search.

Hash method with efficient retrieval efficiency and lower storage space, in the most recent has been widely used in image retrieval task. For complex image retrieval task, the manually extracted features often has limitations, and the deep convolution neural network can extract better features. However, traditional convolution neural networks usually focus on local features, while ignoring the ability to learn global features. In addition, traditional hash methods require constructing sample pairs to learn distance metrics, resulting in high computational costs. To address the above problem, in this paper a new self-attention based deep hash (SADH) learning method is proposed, which introduces a labeled hashing center and trains the network using self-attention mechanism. The method aims to fit the image hash codes to the relevant hashing center. The distance between the hashing center and the image hash codes is calculated using a loss function, resulting in generated hash code with strong discriminative power. Experimental evaluations on standard datasets demonstrate that this method outperforms the state-of-the-art retrieval methods in terms of retrieval accuracy.

Hashing has been widely used for large-scale remote sensing image retrieval due to its outstanding advantages in storage and search speed. Recently, deep hashing methods, which produce discriminative hash codes by building end-to-end deep convolutional networks, have shown promising results. However, training these networks requires numerous labeled images, which are scarce and expensive in remote sensing datasets. In order to solve this problem, we propose a deep unsupervised hashing method, namely deep contrastive self-supervised hashing (DCSH), which uses only unlabeled images to learn accurate hash codes. It eliminates the need for label annotation by maximizing the consistency of different views generated from the same image. More specifically, we assume that the hash codes generated from different views of the same image are similar, and those generated from different images are dissimilar. On the basis of the hypothesis, we can develop a novel loss function containing the temperature-scaled cross-entropy loss and the quantization loss to train the developed deep network end-to-end, resulting in hash codes with semantic similarity preserved. Our proposed network contains four parts. First, each image is transformed into two different views using data augmentation. After that, they are fed into an encoder with the same shared parameters to obtain deep discriminate features. Following this, a hash layer converts the high-dimensional image representations into compact binary codes. Lastly, a novel hash function is introduced to train the proposed network end-to-end and thus guide generated hash codes with semantic similarity. Extensive experiments on two popular benchmark datasets of the UC Merced Land Use Database and the Aerial Image Dataset have demonstrated that our DCSH has significant superiority in remote sensing image retrieval compared with state-of-the-art unsupervised hashing methods.

Recently, image retrieval from massive images has been considered as a crucial research topic. Current deep hashing methods mostly rely on convolutional neural network (CNN) as the backbone network. However, the CNN-based deep hashing methods require multiple convolution operations to capture global information, which lead to the loss of many shallow-level features in the process of feature extraction and affect the accuracy of binary hashing codes. To address such limitations, a new deep hashing method is proposed and studied in this paper by adopting the vision transformer model as the backbone network and designing a spatial reduction layer with channel attention mechanism fused. Such a deep hashing method can capture long-range dependencies effectively. The experiments on the CIFAR-10, NUSWIDE and ImageNet datasets indicate that the proposed method achieves higher mean average precision than other state-of-the-art hashing methods, thereby confirming the outstanding performance on image retrieval.

Content-Based large-scale image retrieval has recently attracted considerable attention because of the explosive increase of online images. Inspired by recent advances in convolutional neural networks, we propose a hierarchical deep semantic method for learning similarity function that solves the problems of precision and speed of retrieval in the setting of large-scale environments. The distinctive contribution of our work is a novel approach that can utilize previous knowledge of the semantic hierarchy. When semantic information and a related hierarchy structure are available, significant improvements can be attained. Exploiting hierarchical relationships is the most important thing for large-scale issues. Binary code can be learned from deep neural network for representing the latent concepts that dominate the semantic labels. Different from other supervised methods that require learning an explicit hashing function to map the binary code features from the images, our method learns Hierarchical Deep Semantic Hashing code (HDSH-code) and image representations in an implicit manner, making it suitable for large-scale datasets. An additional contribution is a novel hashing scheme (generated at the same time with semantic information) that is able to reduce the computational cost of retrieval. Comprehensive experiments were conducted on Holidays, Oxford5k/105k, Caltech256 retrieval datasets, our HDSH performs competitively even when the convolutional neural network has been pre-trained for a surrogate unrelated task. We further demonstrates its efficacy and scalability on a large-scale dataset Imagenet with millions of images. With deep hierarchical semantic hashing, we report retrieval times are 0.15ms and 53.92ms on Holidays and Imagenet dataset, respectively.

Learning valid similarities is a vital problem in hashing methods, especially in large-scale image search. Similarity pertained hashing method is widely used in image retrieval for its high quality compact binary code mapping. The hashing scheme of most existing hashing methods is that the input data is encoded as a vector of visual features and hashed into binary hash codes via projection functions or quantization methods afterward. However, this separated pipeline may prone to lose accurate similarities of images, since the limited compatible domain between visual feature vectors generation and binary codes mapping process. Encouraged by the extraordinary image representation learning ability of deep neural networks in classification, we propose a structure that merges binary code generation process within deep neural networks for efficient image retrieval. The proposed architecture contains two fundamental blocks. The stacked convolution layers of Network In Network with global average pooling compute effective image representation and the embedded latent layer with binary activation functions learn binary hash codes simultaneously. Experiments show that the proposed method gains improvement over several state-of-the-art hashing methods.

Deep flow-net for EPI distortion estimation

It is very crucial for large-scale image retrieval tasks to extract effective hash feature representations. Encouraged by the recent advances in convolutional neural networks (CNNs), this paper presents a novel cascaded deep hashing (CDH) method to generate compact hash codes for highly efficient image retrieval tasks on given large-scale datasets. Specifically, we ingeniously utilize three CNN models to learn robust image feature representations on a given dataset, which solves the issue that categories with poor feature representation have a fairly low retrieval precision. Experimental results indicate that CDH outperforms some state-of-the-art hashing algorithms on both CIFAR-10 and MNIST datasets.

In the field of fast and large-scale image retrieval, the binary hash code based on convolutional features enjoys powerful advantages. CNN model's leading recognition ability shows that its high-level features distribute intensively and have stable between-class distances. On the basis of the distribution characteristic of convolutional features, this paper presents a new method of improving retrieval performance by means of using the category centers of high-level features to enhance the common semantic content of high-level features of the query image and making the consequent binary hash code gain better retrieval results. Experimental results indicate that (i) the category centers of CNN feature offer comparable discriminability with the initial model when identifying the semantic information of images; (ii) before calculating binary hash code, the integration of the CNN feature of a query image with category centers weighted by predicted probabilities can boost performance in similar image retrieval. State-of-the-art retrieval performance is obtained on the Cifar-10 dataset in this paper among all hash methods.

Both 3D models and 2D images contain a wealth of information about everyday objects in our environment. However, it is difficult to semantically link together these two media forms, even when they feature identical or very similar objects. We propose a joint embedding space populated by both 3D shapes and 2D images of objects, where the distances between embedded entities reflect similarity between the underlying objects. This joint embedding space facilitates comparison between entities of either form, and allows for cross-modality retrieval. We construct the embedding space using 3D shape similarity measure, as 3D shapes are more pure and complete than their appearance in images, leading to more robust distance metrics. We then employ a Convolutional Neural Network (CNN) to "purify" images by muting distracting factors. The CNN is trained to map an image to a point in the embedding space, so that it is close to a point attributed to a 3D model of a similar object to the one depicted in the image. This purifying capability of the CNN is accomplished with the help of a large amount of training data consisting of images synthesized from 3D shapes. Our joint embedding allows cross-view image retrieval, image-based shape retrieval, as well as shape-based image retrieval. We evaluate our method on these retrieval tasks and show that it consistently out-performs state-of-the-art methods, and demonstrate the usability of a joint embedding in a number of additional applications.

Image retrieval is a prominent subject of study in the fields of image processing and computer vision. With its application in various domains, such as logo search, product search or general image search in Google, Bing, etc., image retrieval has received significant attention for many years. In this work, we study and investigate a framework that leverages visual transferring fea-tures and hashing algorithms for the purpose of finding similar images in a dataset. The key idea of our solution is to find the answer to the following question: “How can we convert an image into binary code and search it more efficiently in a large-scale dataset?”. To achieve this pur-pose, we use pretrained CNN models from ImageNet for image representation and then convert them into binary code by using hashing algorithms. These images in the dataset are represented by binary codes, and the Hamming distance is used to find the images in the dataset that are in-dexed. To demonstrate the robustness of the system, we systematically tested the performance of the system based on speed with raw indexing and hashing indexing on 4 datasets: CIFAR-10, Caltech-101, Oxford-102-Flowers, and MS-COCO 2017. The experimental results show that lo-cal sensitive hashing (LSH) algorithms with 2,048 bits in binary code demonstrate the same or greater precision than raw indexing. Furthermore, the findings show that the MobileNet architec-ture consistently outperforms other architectures across these datasets, effectively balancing speed and precision.