Rotation Invariant Aerial Image Retrieval with Group Convolutional Metric Learning

Hashing has been widely used for large-scale remote sensing image retrieval due to its outstanding advantages in storage and search speed. Recently, deep hashing methods, which produce discriminative hash codes by building end-to-end deep convolutional networks, have shown promising results. However, training these networks requires numerous labeled images, which are scarce and expensive in remote sensing datasets. In order to solve this problem, we propose a deep unsupervised hashing method, namely deep contrastive self-supervised hashing (DCSH), which uses only unlabeled images to learn accurate hash codes. It eliminates the need for label annotation by maximizing the consistency of different views generated from the same image. More specifically, we assume that the hash codes generated from different views of the same image are similar, and those generated from different images are dissimilar. On the basis of the hypothesis, we can develop a novel loss function containing the temperature-scaled cross-entropy loss and the quantization loss to train the developed deep network end-to-end, resulting in hash codes with semantic similarity preserved. Our proposed network contains four parts. First, each image is transformed into two different views using data augmentation. After that, they are fed into an encoder with the same shared parameters to obtain deep discriminate features. Following this, a hash layer converts the high-dimensional image representations into compact binary codes. Lastly, a novel hash function is introduced to train the proposed network end-to-end and thus guide generated hash codes with semantic similarity. Extensive experiments on two popular benchmark datasets of the UC Merced Land Use Database and the Aerial Image Dataset have demonstrated that our DCSH has significant superiority in remote sensing image retrieval compared with state-of-the-art unsupervised hashing methods.

As a retrieval method, Hash is widely used in the field of image retrieval because it can meet the retrieval time requirements for large-scale image retrieval. In order to solve the problem that most of the current image retrieval models are weak in representing remote sensing images, especially for remote sensing images with complex background, it is easy to confuse the background with the image subject. This paper proposes a remote sensing image retrieval model based on DenseNet. On the ground of original DenseNet model, the method in this paper adds CBAM to each Dense Block of the model, extracts the image features with regional attention and channel attention, and adopts the triple loss function to achieve the retrieve of remote sensing images. The experiment results show the retrieval accuracy of this presented method is significantly improved based on global features in the common datasets of CIFAR-10 and NUS-WIDE as well as the remote sensing image dataset of UC-Merced compared with the existing retrieval methods.

With the increasing amount of high-resolution remote sensing images, large-scale remote sensing image retrieval(RSIR) becomes more and more significant and has attracted great attention. Traditional image retrieval methods generally use hand-crafted features which are not only time-consuming but also always get poor performance. Deep learning recently achieves remarkable performance due to its powerful ability to learn high-level semantic features, so researchers attempt to take advantage of features derived from Convolutional Neural Networks(CNNs) in RSIR. But remote sensing image is different from natural scene image, its background is more complicated with a lot of noise and existing deep learning method didn't handle this well. Both the speed and the accuracy achieve unsatisfactory performance. In this paper, we propose a rotation invariant hashing network that represents an image as a binary hash code to retrieve image faster while considering the rotation invariance of the same target. The results of the experiments on some available remote sensing datasets show that our method is effective and outperforms than other features which is usually used in RSIR.

Since the properties of temporal and spatial complexity and mass diversity that remote sensing image data owns, remote sensing image retrieval becomes an international advanced frontier scientific issue in remote sensing. Content-based image retrieval technology is currently widely used; however, the difference between low-level features and high-level semantics, named semantic gap, becomes a difficult while important issue for remote sensing image retrieval. In this paper, a novel semantic retrieval method for remote sensing images using association rules mining is presented. Unlike the traditional content-based image retrieval methods, association rules are mined and used to express the semantic information of images instead of low-level features. The original image is firstly segmented into many objects; and then the classified association rules between the properties of objects are mined and transformed to semantic information by semantic annotation method; finally the semantic retrieval is achieved using the similarity measurement approach. The experimental results indicate that the proposed method can provide better retrieval performance than the existing content-based image retrieval methods.

Due to the dramatically increased volume of remote sensing (RS) image archives, images are usually stored in a compressed format to reduce the storage size. Existing content-based RS image retrieval (CBIR) systems require as input fully decoded images, thus resulting in a computationally demanding task in the case of large-scale CBIR problems. To overcome this limitation, in this article, we present a novel CBIR system that achieves a coarse-to-fine progressive RS image description and retrieval in the partially decoded Joint Photographic Experts Group (JPEG) 2000 compressed domain. The proposed system initially: 1) decodes the code blocks associated only to the coarse wavelet resolution and 2) discards the most irrelevant images to the query image based on the similarities computed on the coarse resolution wavelet features of the query and archive images. Then, the code blocks associated with the subsequent resolution of the remaining images are decoded and the most irrelevant images are discarded by computing similarities considering the image features associated with both resolutions. This is achieved by using the pyramid match kernel similarity measure that assigns higher weights to the features associated with the finer wavelet resolution than to those related to the coarse wavelet resolution. These processes are iterated until the codestreams associated with the highest wavelet resolution are decoded. Then, the final retrieval is performed on a very small set of completely decoded images. Experimental results obtained on two benchmark archives of aerial images point out that the proposed system is much faster while providing a similar retrieval accuracy than the standard CBIR systems.

A considerable volume of high-resolution remote sensing (HRRS) data is generated with the intense space explorations happening globally. Remote sensing image retrieval (RSIR) is a fundamental task in the remote sensing domain, providing excellent opportunities for a broad spectrum of applications. Difficulty in describing the heterogeneous remote sensing image (RSI) content and the availability in huge volumes is challenging in many analyses and certainly needs many added processing challenges. Also, the popularity of using deep learning (DL) techniques is also increased in the remote sensing domain. The practice of using DL methods to strengthen the efficiency of RSIR frameworks has a very broad prospect. Thus, HRRS data brings new challenges for deep-learning-based RSIR (DL-RSIR) tasks, especially when the data is experienced as ‘big data.’ This article systematically reviews many existing works, concentrating on the advancements and current trends related to DL-RSIR. Also, it narrates the challenges incorporated with the RSIR tasks and how to utilize DL techniques and frameworks to address them. Almost all potential factors that could influence the DL-RSIR performance are analysed, and many evaluations are performed with a comparative analysis. The observations and recommendations presented can help researchers to bring new insights into designing DL-RSIR frameworks.

Benchmark datasets are essential for developing and evaluating remote sensing image retrieval (RSIR) approaches. However, most of the existing datasets are single-labeled, with each image in these datasets being annotated by a single label representing the most significant semantic content of the image. This is sufficient for simple problems, such as distinguishing between a building and a beach, but multiple labels and sometimes even dense (pixel) labels are required for more complex problems, such as RSIR and semantic segmentation.We therefore extended the existing multi-labeled dataset collected for multi-label RSIR and presented a dense labeling remote sensing dataset termed "DLRSD". DLRSD contained a total of 17 classes, and the pixels of each image were assigned with 17 pre-defined labels. We used DLRSD to evaluate the performance of RSIR methods ranging from traditional handcrafted feature-based methods to deep learning-based ones. More specifically, we evaluated the performances of RSIR methods from both single-label and multi-label perspectives. These results demonstrated the advantages of multiple labels over single labels for interpreting complex remote sensing images. DLRSD provided the literature a benchmark for RSIR and other pixel-based problems such as semantic segmentation.

Aerial scene classification is an active and challenging problem in high-resolution remote sensing imagery understanding. Deep learning models, especially convolutional neural networks (CNNs), have achieved prominent performance in this field. The extraction of deep features from the layers of a CNN model is widely used in these CNN-based methods. Although the CNN-based approaches have obtained great success, there is still plenty of room to further increase the classification accuracy. As a matter of fact, the fusion with other features has great potential for leading to the better performance of aerial scene classification. Therefore, we propose two effective architectures based on the idea of feature-level fusion. The first architecture, i.e., texture coded two-stream deep architecture, uses the raw RGB network stream and the mapped local binary patterns (LBP) coded network stream to extract two different sets of features and fuses them using a novel deep feature fusion model. In the second architecture, i.e., saliency coded two-stream deep architecture, we employ the saliency coded network stream as the second stream and fuse it with the raw RGB network stream using the same feature fusion model. For sake of validation and comparison, our proposed architectures are evaluated via comprehensive experiments with three publicly available remote sensing scene datasets. The classification accuracies of saliency coded two-stream architecture with our feature fusion model achieve 97.79%, 98.90%, 94.09%, 95.99%, 85.02%, and 87.01% on the UC-Merced dataset (50% and 80% training samples), the Aerial Image Dataset (AID) (20% and 50% training samples), and the NWPU-RESISC45 dataset (10% and 20% training samples), respectively, overwhelming state-of-the-art methods.

With the explosive growth of remote sensing big data, large-scale remote sensing image retrieval (RSIR) has become one of the most challenging tasks in data mining, attracting more and more attention from researchers. In recent years, deep learning has achieved fruitful results in RSIR. However, there are three major problems. The first problem is due to the imbalanced categories in RSIR data sets. Accordingly, a novel sample mining method is proposed. This method is based on top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> “misplaced” samples (Top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> MS) along the top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> nearest neighbor decision boundary, rather than setting boundary threshold of sample mining and reducing the impact of artificial and objective factors on network training as usual. The second problem comes from the inconsistency between the loss reduction direction and the optimization direction. In response, we additionally propose a novel result ranking loss (RRL) according to the retrieval results in a batch. RRL is used as a plug-in and combined with other local losses. As a result, the retrieval accuracy improved significantly. For the third problem, processing a large number of high-resolution images will increase training time and cause network freeze. To solve the problem, this article proposes a global optimization model based on feature space and retrieval results (FSRR). More specifically, FSRR consists of local loss based on feature space and RRL based on retrieval results, and can be optimized in an end-to-end manner. Experimental results show that our proposed algorithm achieves the best performance when compared with other state-of-the-art deep RSIR algorithms on two benchmark data sets.

This paper proposes a remote sensing (RS) image retrieval scheme by using image scene semantic (SS) matching. The low-level image visual features (VFs) are first mapped into multilevel spatial semantics via VF extraction, object-based classification of support vector machines, spatial relationship inference, and SS modeling. Furthermore, a spatial SS matching model that involves the object area, attribution, topology, and orientation features is proposed for the implementation of the sample-scene-based image retrieval. Moreover, a prototype system that uses a coarse-to-fine retrieval scheme is implemented with high retrieval accuracy. Experimental results show that the proposed method is suitable for spatial SS modeling, particularly geographic SS modeling, and performs well in spatial scene similarity matching.

&amp;lt;p&amp;gt;Since the beginning of the 20&amp;lt;sup&amp;gt;th&amp;lt;/sup&amp;gt; century, various countries have carried out expeditions to Antarctica with the aim of claiming territory, reconnaissance as well as capturing aerial images for topographic mapping. Many of these image inventories has since then been forgotten and never used for scientific purposes. We have gained access to a unique dataset of aerial images captured in 1936-1937 as a part of the Norwegian &amp;lt;em&amp;gt;Thorshavn IV &amp;lt;/em&amp;gt;expedition surveying and mapping large parts of the East Antarctic coastline. The images have a stereo overlap of approximate 60% and are digitized using a photogrammetry-grade scanner, enabling us to produce the earliest known digital elevation models and orthophotos of Antarctica.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;Expanding the observational records of Antarctic glaciers are vital for better understanding and modelling how changes in climatic parameters affects the ice. Currently, we know very little about the behaviour of Antarctic glaciers prior to the 1990s, due to a lack of large-scale observations. Several studies has proven the effectiveness of using digitally-scanned historical aerial images in studying ice mass losses of the pre-satellite era, but very few such studies exist for Antarctica. In this study, we explore Norwegian and Australian historical aerial expedition images collected between 1937 and 1997 to extensively expand the records and provide the earliest regional-scale Antarctic glacier records. The images are processed using structure-from-motion photogrammetry, which enables us to construct accurate, high-resolution digital elevation models and orthophotos. By combining expedition images with modern satellite data, we are creating a unique time-series dataset to study the changes of multiple glaciers along the East Antarctic coastline in Mac Robertson and Kemp Land between 1937 and 2017.&amp;lt;/p&amp;gt;

We are witnessing daily acquisition of large amounts of aerial and satellite imagery. Analysis of such large quantities of data can be helpful for many practical applications. In this letter, we present an automatic content-based analysis of aerial imagery in order to detect and mark arbitrary objects or regions in high-resolution images. For that purpose, we proposed a method for automatic object detection based on a convolutional neural network. A novel two-stage approach for network training is implemented and verified in the tasks of aerial image classification and object detection. First, we tested the proposed training approach using UCMerced data set of aerial images and achieved accuracy of approximately 98.6%. Second, the method for automatic object detection was implemented and verified. For implementation on GPGPU, a required processing time for one aerial image of size 5000 $\times$ 5000 pixels was around 30 s.

This paper presents a new method for building detection and reconstruction from aerial images. In our approach, we extract useful building location information from the generated disparity map to segment the interested objects and consequently reduce unnecessary line segments extracted in the low level feature extraction step. Hypothesis selection is carried out by using an undirected graph, in which close cycles represent complete rooftops hypotheses. We test the proposed method with the synthetic images generated from Avenches dataset of Ascona aerial images. The experiment result shows that the extracted 3D line segments of the reconstructed buildings have an average error of 1.69m and our method can be efficiently used for the task of building detection and reconstruction from aerial images.

Land-cover and land-use classification from aerial images is a challenging problem due to high intra-class diversity and inter-class similarities of the images. To analyze the performances of deep convolutional neural network (CNN) models in this domain, we provide three pre-trained CNN models that are adapted to NWPU-RESISC45 dataset using three different training splits, i.e., 80%, 20%, and 10% ratios. The architectures of all three models are redesigned to be modest in size and their structure is kept simple, yet when tested with the NWPU-RESISC45 dataset, all three models perform comparably to the state-of-the-art models. Each of these models is then used to classify the scenes taken from five well-known datasets in this domain without any fine-tuning. We aim to assess the generalization capabilities of these models on the selected datasets. For better analysis, we considered using top-3 and top-5 accuracies of the models in addition to the best predicted category (top-1) that is usually reported by the models. This way of interpretation is very suitable in this domain, since the datasets contain a high number of fine grained categories with large semantic overlaps. We empirically show that the proposed CNN models actually learn the relevant semantic features in the aerial images better than we observed via standard measures. To the best of our knowledge, this is the first work in this domain that analyzes and presents model generalization performances the way we presented it here.

Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark