Scene classification in compressed and constrained domain

Scene classification plays an important role in the interpretation of remotely sensed high-resolution imagery. However, the performance of scene classification strongly relies on the discriminative power of feature representation, which is generally hand-engineered and requires a huge amount of domain-expert knowledge as well as time-consuming hand tuning. Recently, unsupervised feature learning (UFL) provides an alternative way to automatically learn discriminative feature representation from images. However, the performances achieved by conventional UFL methods are not comparable to the state-of-the-art, mainly due to the neglect of locally substantial image structures. This paper presents an improved UFL algorithm based on spectral clustering, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">named</i> UFL-SC, which cannot only adaptively learn good local feature representations but also discover intrinsic structures of local image patches. In contrast to the standard UFL pipeline, UFL-SC first maps the original image patches into a low-dimensional and intrinsic feature space by linear manifold analysis techniques, and then learns a dictionary (e.g., using K-means clustering) on the patch manifold for feature encoding. To generate a feature representation for each local patch, an explicit parameterized feature encoding method, i.e., triangle encoding, is applied with the learned dictionary on the same patch manifold. The holistic feature representation of image scenes is finally obtained by building a bag-of-visual-words (BOW) model of the encoded local features. Experiments demonstrate that the proposed UFL-SC algorithm can extract efficient local features for image scenes and show comparable performance to the state-of-the-art approach on open scene classification benchmark.

This study aims to compare the integrity and reproducibility of measurements created from uncompressed and compressed digital images in order to implement compliance with 21 CFR Part 11 for image analysis studies executed using 21 CFR Part 58 compliant capture systems. Images of a 400-mesh electron microscope grid and H&E stained rat liver tissue were captured on an upright microscope with digital camera using commercially available analysis software. Digital images were stored as either uncompressed TIFFs or in one of five different levels of JPEG compression. The grid images were analyzed with automatic detection of bright objects while the liver images were segmented using color cube-based morphometry techniques, respectively, using commercially-available image analysis software. When comparing the feature-extracted measurements from the TIFF uncompressed to the JPEG compressed images, the data suggest that JPEG compression does not alter the accuracy or reliability to reproduce individual data point measurements in all but the highest compression levels. There is, however, discordance if the initial measure was obtained with a TIFF format and subsequently saved as one of the JPEG levels, suggesting that the use of compression must precede feature extraction. It is a common practice in software packages to work with TIFF uncompressed images. However, this study suggests that the use of JPEG compression as part of the analysis work flow was an acceptable practice for these images and features. Investigators applying image file compression to other organ images will need to validate the utility of image compression in their work flow. A procedure to digitally acquire and JPEG compress images prior to image analysis has the potential to reduce file archiving demands without compromising reproducibility of data.

By labelling high spatial resolution (HSR) images with specific semantic classes according to geographical properties, scene classification has been proven to be an effective method for HSR remote sensing image semantic interpretation. Deep learning is widely applied in HSR remote sensing scene classification. Most of the scene classification methods based on deep learning assume that the training datasets and the test datasets come from the same datasets or obey similar feature distributions. However, in practical application scenarios, it is difficult to guarantee this assumption. For new datasets, it is time-consuming and labor-intensive to repeat data annotation and network design. The neural architecture search (NAS) can automate the process of redesigning the baseline network. However, traditional NAS lacks the generalization ability to different settings and tasks. In this paper, a novel neural network search architecture framework—the spatial generalization neural architecture search (SGNAS) framework—is proposed. This model applies the NAS of spatial generalization to cross-domain scene classification of HSR images to bridge the domain gap. The proposed SGNAS can automatically search the architecture suitable for HSR image scene classification and possesses network design principles similar to the manually designed networks. To obtain a simple and low-dimensional search space, the traditional NAS search space was optimized and the human-the-loop method was used. To extend the optimized search space to different tasks, the search space was generalized. The experimental results demonstrate that the network searched by the SGNAS framework with good generalization ability displays its effectiveness for cross-domain scene classification of HSR images, both in accuracy and time efficiency.

Remote sensing (RS) scene classification has been given important consideration due to its wide-ranging usage by RS communal. Scene classification in satellite images has extensive usage in environmental observation, remote scene study, and surveillance. Due to the huge improvements of scene classification task, different techniques were recommended in existing times for automated categorization of land scenes from RS images (RSIs). It plays a main part in a wide range of applications, namely geospatial object detection, environment monitoring, natural hazards detection, and so on. Recently, machine learning (ML) and Artificial Intelligence (AI) have accomplished considerable successes and are newly initiated in detection of remote. Moreover, the deep learning (DL) have attained high efficacy in classification of image, and the precision is always being continued with the growth of new methods. This paper presents an Artificial Intelligence Driven Deep Learning Method for Scene Classification in Remote Sensing Images (AIDLM-SCRSI) technique. The main intention of AIDLM-SCRSI technique relies on improving the remote sensing scene classification using advanced DL techniques. At first, the image pre-processing stage applies wiener filtering (WF) to eliminate the unwanted noise in input image data. Next, the proposed AIDLM-SCRSI method designs MobileNetV2 method for the extraction of feature process. Furthermore, the conventional autoencoder (CAE) technique has been implemented for the scene classification process. Eventually, the zebra optimization algorithm (ZOA) fine-tunes the hyperparameter values of the CAE algorithm optimally leading to superior performance of the classification. A comprehensive performance was implemented to verify the performance of the AIDLM-SCRSI approach. The stimulated outcome identified that the AIDLM-SCRSI model underlined improvement over other recent techniques.

Image understanding and scene classification are keystone tasks in computer vision. The development of technologies and profusion of existing datasets open a wide room for improvement in the image classification and recognition research area. Notwithstanding the optimal performance of exiting machine learning models in image understanding and scene classification, there are still obstacles to overcome. All models are data-dependent that can only classify samples close to the training set. Moreover, these models require large data for training and learning. The first problem is solved by few-shot learning, which achieves optimal performance in object detection and classification but with a lack of eligible attention in the scene classification task. Motivated by these findings, in this paper, we introduce two models for few-shot learning in scene classification. In order to trace the behavior of those models, we also introduce two datasets (MiniSun; MiniPlaces) for image scene classification. Experimental results show that the proposed models outperform the benchmark approaches in respect of classification accuracy.

A learning approach to knowledge-assisted image analysis and classification is proposed that combines global and local information with explicitly defined knowledge in the form of an ontology. The ontology specifies the domain of interest, its subdomains, the concepts related to each subdomain as well as contextual information. Support vector machines (SVMs) are employed in order to provide image classification to the ontology subdomains based on global image descriptions. In parallel, a segmentation algorithm is applied to segment the image into regions and SVMs are again employed, this time for performing an initial mapping between region low-level visual features and the concepts in the ontology. Then, a decision function, that receives as input the computed region-concept associations together with contextual information in the form of concept frequency of appearance, realizes image classification based on local information. A fusion mechanism subsequently combines the intermediate classification results, provided by the local- and global-level information processing, to decide on the final image classification. Once the image subdomain is selected, final region-concept association is performed using again SVMs and a genetic algorithm (GA) for optimizing the mapping between the image regions and the selected subdomain concepts taking into account contextual information in the form of spatial relations. Application of the proposed approach to images of the selected domain results in their classification (i.e., their assignment to one of the defined subdomains) and the generation of a fine granularity semantic representation of them (i.e., a segmentation map with semantic concepts attached to each segment). Experiments with images from the personal collection domain, as well as comparative evaluation with other approaches of the literature, demonstrate the performance of the proposed approach.

Effective scene representation is a fundamental part of high-resolution scene classification systems. In this letter, we present a holistic scene representation method, i.e., the two-level feature representation (TLFR) model. The TLFR is composed of low-level and high-level features. Low-level features are obtained by computing the residual error between a local descriptor and its corresponding visual word, and the high-level features are obtained using a proposed selection-constrained sparse coding method. In addition, low-level features in a cluster are integrated by summation pooling, whereas high-level features are fused by maximization pooling. The holistic scene representation is finally generated by incorporating these two levels of features into the bag-of-visual-words framework. Experimental results show that the TLFR model is robust to translation and rotation variations and demonstrates promising performance with the Land Use and Land Cover Database data set and a newly released Singapore data set.

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Scene classification in remote sensing is a pivotal research area, traditionally relying on visual information from aerial images for labeling. The introduction of ground environment audio as a novel geospatial data source adds valuable information for scene classification. However, bridging the structural gap between aerial images and ground environment audio is challenging, rendering popular two-branch networks ineffective for direct data fusion. To address this issue, the study in this research presents the Two-stage Fusion-based Audiovisual Classification Network (TFAVCNet). TFAVCNet leverages both audio and visual modules to extract deep semantic features from ground environmental audio and remote sensing images, respectively. The audiovisual fusion module combines and fuses information from both modalities at the feature and decision levels, facilitating joint training and yielding a more-robust solution. The proposed method outperforms existing approaches, as demonstrated by the experimental results on the ADVANCE dataset for remote sensing audiovisual scene classification, offering an innovative approach to enhanced scene classification.

Late fusion framework for Acoustic Scene Classification using LPCC, SCMC, and log-Mel band energies with Deep Neural Networks

Millions of digital images are captured by imaging devices on a daily basis. The way imaging devices operate follows an integral process from which the information of the original scene needs to be estimated. The estimation is done by inverting the integral process of the imaging device with the use of optimization techniques. This linear inverse problem, the inversion of the integral acquisition process, is at the heart of several image processing applications such as denoising, deblurring, inpainting, and super-resolution. We describe in detail the use of linear inverse problems in these applications. We review and compare several state-of-the-art optimization algorithms that invert this integral process. Linear inverse problems are usually very difficult to solve. Therefore, additional prior assumptions need to be introduced to successfully estimate the output signal. Several priors have been suggested in the research literature, with the Total Variation (TV) being one of the most prominent. In this thesis, we review another prior, the l0 pseudo-norm over the gradient domain. This prior allows full control over how many non-zero gradients are retained to approximate prominent structures of the image. We show the superiority of the l0 gradient prior over the TV prior in recovering genuinely piece-wise constant signals. The l0 gradient prior has shown to produce state-of-the-art results in edge-preserving image smoothing. Moreover, this general prior can be applied to several other applications, such as edge extraction, clip-art JPEG artifact removal, non-photorealistic image rendering, detail magnification, and tone mapping. We review and evaluate several state-of-the-art algorithms that solve the optimization problem based on the l0 gradient prior. Subsequently we apply the l0 gradient prior to two applications where we show superior results as compared to the current state-of-the-art. The first application is that of single-image reflection removal. Existing solutions to this problem have shown limited success because of the highly ill-posed nature of the problem. We show that the standard l0 gradient prior with a modified data-fidelity term based on the Laplacian operator is able to sufficiently remove unwanted reflections from images in many realistic scenarios. We conduct extensive experiments and show that our method outperforms the state-of-the-art. In the second application of haze removal from visible-NIR image pairs we propose a novel optimization framework, where the prior term penalizes the number of non-zero gradients of the difference between the output and the NIR image. Due to the longer wavelengths of NIR, an image taken in the NIR spectrum suffers significantly less from haze artifacts. Using this prior term, we are able to transfer details from the haze-free NIR image to the final result. We show that our formulation provides state-of-the-art results compared to haze removal methods that use a single image and also to those that are based on visible-NIR image pairs.

Effective representation and classification of scenes using very high resolution (VHR) remote sensing images cover a wide range of applications. Although robust low‐level image features have been proven to be effective for scene classification, they are not semantically meaningful and thus have difficulty to deal with challenging visual recognition tasks. In this study, the authors propose a new and effective auto‐encoder‐based method to learn a shared mid‐level visual dictionary. This dictionary serves as a shared and universal basis to discover mid‐level visual elements. On the one hand, the mid‐level visual dictionary learnt using machine learning technique is more discriminative and contains rich semantic information, compared with the traditional low‐level visual words. On the other hand, the mid‐level visual dictionary is more robust to occlusions and image clutters. In the authors' scene‐classification scheme, they use discriminative mid‐level visual elements, rather than individual pixels or low‐level image features, to represent images. This new image representation is able to capture much of the high‐level meaning and contents of the image, facilitating challenging remote sensing image scene‐classification tasks. Comprehensive evaluations on a challenging VHR remote sensing images data set and comparisons with state‐of‐the‐art approaches demonstrate the effectiveness and superiority of their study.

To aid an automatic taxiing system for unmanned aircraft, this paper presents a colour based method for semantic segmentation and image classification in an aerodrome environment with the intention to use the classification output to aid navigation and collision avoidance. Based on previous work, this machine vision system uses semantic segmentation to interpret the scene. Following an initial superpixel based segmentation procedure, a colour based Bayesian Network classifier is trained and used to semantically classify each segmented cluster. HSV colourspace is adopted as it is close to the way of human vision perception of the world, and each channel shows significant differentiation between classes. Luminance is used to identify surface lines on the taxiway, which is then fused with colour classification to give improved classification results. The classification performance of the proposed colour based classifier is tested in a real aerodrome, which demonstrates that the proposed method outperforms a previously developed texture only based method.

Understanding the context of the scene is one of the most important aspects for new generation of autonomous vehicles. It is a very trivial task for a human to recognize the scene context by only a single look at the picture, however, for a computer, it is still a challenging task. This problem can be solved by automatic data labeling using deep-learning models for scene classification. Relying on scene type labels we can select relevant scenes to prepare a balanced dataset to train more advanced instance detection models using data from a specific road condition. This study presents a novel framework based on a deep convolutional neural network (CNN) for the automatic road scene classification of street-level automotive images. For the evaluation of our approach, we use a well-known autonomous benchmark dataset, from which we extract geo-position data and combine them with predictions from the scene classification model to get ground truth labels to train and evaluate a ResNet-50 model for scene classification. The results and comparison with state-of-the-art methods are presented.

Which is the best way to organize/classify images by content?