Ensemble Learning for Multispectral Scene Classification

In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary in low light condition, the multispectral fusion of RGB and NIR images provides a viable solution to low light imaging. In this paper, we propose multispectral fusion of RGB and NIR images using weighted least squares (WLS) and convolution neural networks (CNNs). We combine traditional WLS filtering for layer decomposition and denoising with latest deep learning for image enhancement and texture transfer into the multispectral fusion to take both advantages. We build two networks based on CNN: image enhancement network (IEN) for image enhancement and texture transfer network (TTN) for NIR texture transfer. First, we perform RGB image denoising based on WLS filtering and generate the base layer. We use both RGB and NIR images for WLS filtering as weights to filter out noise in low light RGB images. Second, we conduct IEN to enhance contrast of the base layer. Third, we perform TTN to deliver NIR details completely and naturally to the fusion result. The combination of WLS, TTN and IEN leads to noise reduction, contrast enhancement, and detail preservation in fusion. Experimental results show that the proposed method achieves good performance in both noise reduction and detail transfer as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.

Scene classification of high resolution remote sensing images plays an important role for a wide range of applications. While significant efforts have been made in developing various methods for scene classification, most of them are based on handcrafted or shallow learning-based features. In this paper, we investigate the use of deep convolutional neural network (CNN) for scene classification. To this end, we first adopt two simple and effective strategies to extract CNN features: (1) using pre-trained CNN models as universal feature extractors, and (2) domain-specifically fine-tuning pre-trained CNN models on our scene classification dataset. Then, scene classification is carried out by using simple classifiers such as linear support vector machine (SVM). In our work, three off-the-shelf CNN models including AlexNet [1], VGGNet [2], and GoogleNet [3] are investigated. Comprehensive evaluations on a publicly available 21 classes land use dataset and comparisons with several state-of-the-art approaches demonstrate that deep CNN features are effective for scene classification of high resolution remote sensing images.

Remote sensing image (RSI) scene classification is the foundation and important technology of ground object detection, land use management and geographic analysis. During recent years, convolutional neural networks (CNNs) have achieved significant success and are widely applied in RSI scene classification. However, crafted images that serve as adversarial examples can potentially fool CNNs with high confidence and are hard for human eyes to interpret. For the increasing security and robust requirements of RSI scene classification, the adversarial example problem poses a serious problem for the classification results derived from systems using CNN models, which has not been fully recognized by previous research. In this study, to explore the properties of adversarial examples of RSI scene classification, we create different scenarios by testing two major attack algorithms (i.e., the fast gradient sign method (FGSM) and basic iterative method (BIM)) trained on different RSI benchmark datasets to fool CNNs (i.e., InceptionV1, ResNet and a simple CNN). In the experiment, our results show that CNNs of RSI scene classification are also vulnerable to adversarial examples, and some of them have a fooling rate of over 80%. These adversarial examples are affected by the architecture of CNNs and the type of RSI dataset. InceptionV1 has a fooling rate of less than 5%, which is lower than the others. Adversarial examples generated on the UCM dataset are easier than other datasets. Importantly, we also find that the classes of adversarial examples have an attack selectivity property. Misclassifications of adversarial examples of RSIs are related to the similarity of the original classes in the CNN feature space. Attack selectivity reveals potential classes of adversarial examples and provides insights into the design of defensive algorithms in future research.

Scene classification plays a crucial role in the travel and tourism industry in promoting travel. In this paper, we propose a fusion approach based on representation learning to perform scene classification using a basic convolutional neural network (CNN) and multiple pre-trained CNNs as feature extractors. We have used the Oliva and Torralba Scene (OT-Scene) dataset consisting of scene images of eight different categories. In this proposed approach we extract image representations from deep learning models such as basic CNN and pre-trained CNN, namely VGG19, MobileNetV2, and RestNet50. We then fuse these representations from several models into different combinations to achieve a more generalized representation or feature vectors. The fused feature vectors and the individual feature vectors are then fed into the artificial neural network (ANN), support vector machine (SVM), and random forest (RF) classifiers separately to compare the results. The results showed the highest scene classification accuracy of 97.4% from specific fused combinations.

ABSTRACTIn recent years, object-based segmentation methods and shallow-model classification algorithms have been widely integrated for remote sensing image supervised classification. However, as the image resolution increases, remote sensing images contain increasingly complex characteristics, leading to higher intraclass heterogeneity and interclass homogeneity and thus posing substantial challenges for the application of segmentation methods and shallow-model classification algorithms. As important methods of deep learning technology, convolutional neural networks (CNNs) can hierarchically extract higher-level spatial features from images, providing CNNs with a more powerful recognition ability for target detection and scene classification in high-resolution remote sensing images. However, the input of the traditional CNN is an image patch, the shape of which is scarcely consistent with a given segment. This inconsistency may lead to errors when directly using CNNs in object-based remote sensing classification: jagged errors may appear along the land cover boundaries, and some land cover areas may overexpand or shrink, leading to many obvious classification errors in the resulting image. To address the above problem, this paper proposes an object-based and heterogeneous segment filter convolutional neural network (OHSF-CNN) for high-resolution remote sensing image classification. Before the CNN processes an image patch, the OHSF-CNN includes a heterogeneous segment filter (HSF) to process the input image. For the segments in the image patch that are obviously different from the segment to be classified, the HSF can differentiate them and reduce their negative influence on the CNN training and decision-making processes. Experimental results show that the OHSF-CNN not only can take full advantage of the recognition capabilities of deep learning methods but also can effectively avoid the jagged errors along land cover boundaries and the expansion/shrinkage of land cover areas originating from traditional CNN structures. Moreover, compared with the traditional methods, the proposed OHSF-CNN can achieve higher classification accuracy. Furthermore, the OHSF-CNN algorithm can serve as a bridge between deep learning technology and object-based segmentation algorithms thereby enabling the application of object-based segmentation methods to more complex high-resolution remote sensing images.

Converting near-infrared (NIR) images into color images is a challenging task due to the different characteristics of visible and NIR images. Most methods of generating color images directly from a single NIR image are limited by the scene and object categories. In this paper, we propose a novel approach to recovering object colors from multi-spectral NIR images using gray information. The multi-spectral NIR images are obtained by a 2-CCD NIR/RGB camera with narrow NIR bandpass filters of different wavelengths. The proposed approach is based on multi-spectral NIR images to estimate a conversion matrix for NIR to RGB conversion. In addition to the multi-spectral NIR images, a corresponding gray image is used as a complementary channel to estimate the conversion matrix for NIR to RGB color conversion. The conversion matrix is obtained from the ColorChecker's 24 color blocks using polynomial regression and applied to real-world scene NIR images for color recovery. The proposed approach has been evaluated by a large number of real-world scene images, and the results show that the proposed approach is simple yet effective for recovering color of objects.

Semantic-level land-use scene classification is a challenging problem, in which deep learning methods, e.g., convolutional neural networks (CNNs), have shown remarkable capacity. However, a lack of sufficient labeled images has proved a hindrance to increasing the land-use scene classification accuracy of CNNs. Aiming at this problem, this paper proposes a CNN pre-training method under the guidance of a human visual attention mechanism. Specifically, a computational visual attention model is used to automatically extract salient regions in unlabeled images. Then, sparse filters are adopted to learn features from these salient regions, with the learnt parameters used to initialize the convolutional layers of the CNN. Finally, the CNN is further fine-tuned on labeled images. Experiments are performed on the UCMerced and AID datasets, which show that when combined with a demonstrative CNN, our method can achieve 2.24% higher accuracy than a plain CNN and can obtain an overall accuracy of 92.43% when combined with AlexNet. The results indicate that the proposed method can effectively improve CNN performance using easy-to-access unlabeled images and thus will enhance the performance of land-use scene classification especially when a large-scale labeled dataset is unavailable.

Scene classification of high spatial resolution (HSR) images can provide data support for many practical applications, such as land planning and utilization, and it has been a crucial research topic in the remote sensing (RS) community. Recently, deep learning methods driven by massive data show the impressive ability of feature learning in the field of HSR scene classification, especially convolutional neural networks (CNNs). Although traditional CNNs achieve good classification results, it is difficult for them to effectively capture potential context relationships. The graphs have powerful capacity to represent the relevance of data, and graph-based deep learning methods can spontaneously learn intrinsic attributes contained in RS images. Inspired by the abovementioned facts, we develop a deep feature aggregation framework driven by graph convolutional network (DFAGCN) for the HSR scene classification. First, the off-the-shelf CNN pretrained on ImageNet is employed to obtain multilayer features. Second, a graph convolutional network-based model is introduced to effectively reveal patch-to-patch correlations of convolutional feature maps, and more refined features can be harvested. Finally, a weighted concatenation method is adopted to integrate multiple features (i.e., multilayer convolutional features and fully connected features) by introducing three weighting coefficients, and then a linear classifier is employed to predict semantic classes of query images. Experimental results performed on the UCM, AID, RSSCN7, and NWPU-RESISC45 data sets demonstrate that the proposed DFAGCN framework obtains more competitive performance than some state-of-the-art methods of scene classification in terms of OAs.

Scene classification technology based on computer vision has been widely applied in many fields. However, with the increasing complexity of images, many computer vision classification models are difficult to meet requirements of current scene classification tasks, as they not only require considering the object, background, spatial layout and other information, but also many relationships in the image. Based on the analysis of existing scene classification algorithms and Xception model, an approach that adds optimization from two aspects of data set processing is proposed to solve complicated scene classification tasks. Combined with the image enhancement technology, the serialized image enhancement method is used to expand the dataset and enhance the image features, and takes advantage of the Xception model to extract the image features to obtain the scene classification model with high robustness. The experimental results showed that Xception model was able to deal with scene classification efficiently by making up for the shortcomings of traditional Convolutional Neural Networks (CNN) models in feature extraction and generalization ability.

Understanding the context of the scene is one of the most important aspects for new generation of autonomous vehicles. It is a very trivial task for a human to recognize the scene context by only a single look at the picture, however, for a computer, it is still a challenging task. This problem can be solved by automatic data labeling using deep-learning models for scene classification. Relying on scene type labels we can select relevant scenes to prepare a balanced dataset to train more advanced instance detection models using data from a specific road condition. This study presents a novel framework based on a deep convolutional neural network (CNN) for the automatic road scene classification of street-level automotive images. For the evaluation of our approach, we use a well-known autonomous benchmark dataset, from which we extract geo-position data and combine them with predictions from the scene classification model to get ground truth labels to train and evaluate a ResNet-50 model for scene classification. The results and comparison with state-of-the-art methods are presented.

A multi-level improved circle pooling for scene classification of high-resolution remote sensing imagery

Acoustic scene classification is the task of identifying the scene from which the audio signal is recorded. Convolutional neural network (CNN) models are widely adopted with proven successes in acoustic scene classification. However, there is little insight on how an audio scene is perceived in CNN, as what have been demonstrated in image recognition research. In the present study, the Class Activation Mapping (CAM) is utilized to analyze how the log-magnitude Mel-scale filter-bank (log-Mel) features of different acoustic scenes are learned in a CNN classifier. It is noted that distinct high-energy time-frequency components of audio signals generally do not correspond to strong activation on CAM, while the background sound texture are well learned in CNN. In order to make the sound texture more salient, we propose to apply the Difference of Gaussian (DoG) and Sobel operator to process the log-Mel features and enhance edge information of the time-frequency image. Experimental results on the DCASE 2017 ASC challenge show that using edge enhanced log-Mel images as input feature of CNN significantly improves the performance of audio scene classification.

Remote sensing (RS) scene classification plays an important role in the field of earth observation. With the rapid development of the RS techniques, a large number of RS scene images are available. As manually labeling large-scale RS scene images is both labor and time consuming, when a new unlabeled data set is obtained, how to use the existing labeled data sets to classify the new unlabeled images is an important research direction. Different RS scene image data sets may be taken from different type of sensors, and the images may vary from imaging modalities, spatial resolutions, and image scales, so the distribution discrepancy exists among different image data sets. As a result, simply applying convolutional neural networks (CNN) trained on source domain cannot accurately classify the images on target domain. Domain adaptation (DA) can be helpful to solve this problem. In this letter, we design a subspace alignment (SA) and CNN-based framework to solve the DA problem in RS scene image classification. A new SA layer is proposed and added into CNN models for DA, which could align the source and target domains in some feature subspace. Fine-tuning the modified CNN model with the added SA layer makes the CNN model adapt to the aligned feature subspace and helps to relieve the domain distribution discrepancy. The experiments conducted on two public data sets show that adding the SA layer into CNN improves the scene classification on the target domain.

Railway detection task produces a large number of images, the lack of effective image classification method makes it difficult to analyze detection image deeply. Using convolutional neural networks (CNN) to realize railway image scene classification is an effective technical means. This paper propose a method to reduce the bias of database by Gradient-weighted Class Activation Mapping(Grad-CAM) to effectively improve scene classification accuracy, and achieve accuracy of 95.3%(top3) on Railway12 database. Our approach combines two insights: (1) Small quantity of railway scene database make it hard for CNN to achieve high performance, transfer pre-trained ImageNet-CNN to fine-tune on railway scene database (2) Introduce Grad-CAM visualization method to analyze model’s classification pattern and intuitively show the possible bias of database, provide an intuitive way to reduce bias of dataset.

Convolutional neural networks (CNNs) have become the state-of-the-art in automatic feature learning which has led to outstanding performance on pattern recognition applications. On the other hand, methodologies for discriminant analysis on multiclass problems have been proposed to determine the discriminant contribution of each feature. In this paper, we combine the capabilities of CNN and multi-class discriminant analysis to propose a general data-driven methodology for feature learning and discriminability for texture classification. The whole pipeline has two main blocks: (a) An approach that understands intrinsic patterns in small image patches using CNNs customized for the focused problem; (b) A multi-class discriminant analysis technique, fed with the CNN output, to select the most discriminant features for classification tasks. Our experimental results have shown that the feature spaces generated by the combination of CNN and discriminant analysis allow higher recognition rates using very much less CNN features in five-class granite image analysis