Regional clustering-based spatial preprocessing for hyperspectral unmixing

Pansharpening is an important way of integrating spatial and spectral information in the field of remote sensing. This field uses the complementary and redundant information between multispectral (MS) images and panchromatic (PAN) images to obtain high spectral and high spatial resolution images. Various pansharpening methods have been introduced so far, each one attempting to provide a pansharpened image with the least distortion and maximum preservation of spectral and spatial information. Due to the importance of this issue, there should be methods and indices to evaluate the performance of different pansharpening algorithms and assess the quality of pansharpened images. In this paper, a segmentation-based method for assessing the quality of fused images is proposed. The advantage of this approach over pixel-based methods is that the pixel-based methods consider the fused images as a set of separate pixels while segmentation can take into account useful spatial information such as neighbourhoods, textures, etc. In the proposed method, by using k-means clustering algorithm, the reference and pansharpened images are segmented into areas with similar spectral and spatial features and the corresponding segments of the images are compared. This method is tested on three real data sets acquired by Pleiades, GeoEye-1, and QuickBird sensors. Experimental results demonstrate the effectiveness of the proposed method in evaluation of the quality of fused images.

The fusion of traditional HSI mainly involves the fusion of HSI with MSI or PAN. Nevertheless, the combination of the former is constrained by the inadequate spatial resolution of MSI, and therefore the enhancement of the spatial resolution of HSI is inadequate. The latter is constrained by the substantial disparity in spectral information between PAN and HSI, leading to pronounced distortion of high-resolution HSI. To address this issue, this paper proposes IFNPMH, which is an integrated fusion network for used for the fusion of HSI, MSI, and PAN. The IFNPMH algorithm is capable of efficiently extracting spatial information from MSI and PAN, as well as spectral information from HSI. It combines the spatial and spectral information to produce fused images that possess precise spectral information and distinct spatial details. Specifically, IFNPMH incorporates the CBAM attention mechanism module to enhance feature representation capabilities. Finally, this article proposes a loss function that combines spectral and spatial information loss in IFNPMH training. The effectiveness of the proposed IFNPMH is demonstrated by extensive testing in comparison to other sophisticated approaches. This model has shown promising performance in terms of qualitative visual effects and quantitative evaluation metrics.

Linear spectral unmixing is an essential technique in hyperspectral image (HSI) processing and interpretation. In recent years, deep learning-based approaches have shown great promise in hyperspectral unmixing (HU), in particular, unsupervised unmixing methods based on autoencoder (AE) networks are a recent trend. The AE model, which automatically learns low-dimensional representations (abundances) and reconstructs data with their corresponding bases (endmembers), has achieved superior performance in HU. In this article, we explore the effective utilization of spatial and spectral information in AE-based unmixing networks. Important findings on the use of spatial and spectral information in the AE framework are discussed. Inspired by these findings, we propose a spatial–spectral collaborative unmixing network, called SSCU-Net, which learns a spatial AE network and a spectral AE network in an end-to-end manner to more effectively improve the unmixing performance. SSCU-Net is a two-stream deep network and shares an alternating architecture, where the two AE networks are efficiently trained in a collaborative way for estimation of endmembers and abundances. Meanwhile, we propose a new spatial AE network by introducing a superpixel segmentation method based on abundance information, which greatly facilitates the employment of spatial information and improves the accuracy of unmixing network. Moreover, extensive ablation studies are carried out to investigate the performance gain of SSCU-Net. Experimental results on both synthetic and real hyperspectral datasets illustrate the effectiveness and competitiveness of the proposed SSCU-Net compared with several state-of-the-art HU methods.

Hyperspectral unmixing (HSU) is a crucial method to determine the fractional abundance of the material (endmembers) in each pixel. Most spectral unmixing methods are affected by low signal-to-noise ratios because of noisy pixels and bands simultaneously, requiring robust HSU techniques that exploit both 3D (spectral–spatial dimension) and 2D (spatial dimension) domains. In this paper, we present a new method for robust supervised HSU based on a deep hybrid (3D and 2D) convolutional autoencoder (DHCAE) network. Most HSU methods adopt the 2D model for simplicity, whereas the performance of HSU depends on spectral and spatial information. The DHCAE network exploits spectral and spatial information of the remote sensing images for abundance map estimation. In addition, DHCAE uses dropout to regularize the network for smooth learning and to avoid overfitting. Quantitative and qualitative results confirm that our proposed DHCAE network achieved better hyperspectral unmixing performance on synthetic and three real hyperspectral images, i.e., Jasper Ridge, urban and Washington DC Mall datasets.

ABSTRACTHyperspectral image feature extraction generally does not consider the optimisation of structure element types, resulting in the loss of spatial correlation information. To solve this issue, this paper proposes a novel adaptive classification method (MPDTNC-SVM), in which spatial information is extracted by Morphological Profile Filter (MPF) and Domain Transform Normalised Convolution Filter (DTNCF). First, MPF extracts the spatial features on multiple principal component analysis (PCA) components of the hyperspectral image, and DTNCF works over all spectral bands to extract spatially correlated features. The two spatial features are then combined and fed into Support Vector Machine (SVM). Second, a two-step optimisation is implemented in the classification process. Specifically, the best structure of MPF is chosen for classification. Next, the optimal parameters of the structural elements are obtained through iterative classification optimisation, with the best classification performance produced in the process. Experimental results of actual hyperspectral images show that the proposed MPF and DTNCF with SVM (MPDTNC-SVM) method is superior to other classification methods, including Edge-Preserving Filter and Recursive Filter method, morphological feature-based methods, and the SVM methods with raw spectral features, reduced-dimensional and spatial-spectral information.

Spectral unmixing is an important problem for remotely sensed hyperspectral data exploitation. Automatic spectral unmixing can be viewed as a three-stage problem, where the first stage is subspace identification, the next one is endmember extraction, and the final one is abundance estimation. In this sequence, endmember extraction is the most challenging problem. Many researchers have attempted to extract endmembers from hyperspectral images using spectral information only. However, it is well known that the inclusion of spatial information can improve the endmember extraction task. In this article, we introduce a new endmember extraction algorithm that exploits both spectral and spatial information. A main innovation of the proposed algorithm is that spatial information is exploited using entropy, while spectral information is exploited using convex set optimization. In the literature, none of the spatial–spectral algorithms has used entropy as spatial information. The inclusion of this entropy-based spatial information improves the accuracy of the endmember extraction process. The results obtained by the proposed algorithm are compared (using a variety of metrics) with those obtained by other state-of-the-art methods, using both synthetic and real datasets. Our experimental results demonstrate that the proposed algorithm outperforms many available algorithms.

Leaves are the primary interface where energy, water and carbon exchanges occur between the forest ecosystems and the atmosphere. Leaf area index (LAI) is a measure of the amount of leaf area in a stand, and the tree crown size characterizes how leaves are clumped in the canopy. Both LAI and tree crown size are of essential ecological and management value. There is a lot of interest in extracting both canopy structural parameters from remote sensing. The LAI is generally estimated with spectral information from remotely sensed images at relatively coarse spatial resolution. There has been much less success in estimating tree crown size with remote sensing. The recent availability of abundant high spatial resolution imagery from space offers new potential for extracting LAI and tree crown size, particularly in the spatial domain. This study found that the spatial information in Ikonos imagery is highly valuable in estimating both tree crown size and LAI. When the conifer‐ and hardwood‐dominated stands are pooled, tree crown sizes of conifer stands relate best to the ratio of image variance at 2×2 m spatial resolution to that at 3×3 m spatial resolution, while LAI relates best to image variance at 4×4 m spatial resolution. When the conifer‐ and hardwood‐dominated stands are separated, image spatial information estimates tree crown size much better for conifer‐dominated stands than for the hardwood‐dominated stands, while the relationship between image spatial information and LAI is strengthened after the two types of stands are combined. Tree crown size is more sensitive to image spatial resolution than LAI. Image variance is more useful in estimating LAI than normalized difference vegetation index (NDVI) and simple ratio vegetation index (SRVI). Combining both spatial and spectral information provides some improvement in estimating LAI compared with using spatial information alone. Therefore, future efforts to estimate canopy structure with high resolution imagery should also use image spatial information.

The pansharpening algorithm often faces an imbalance between spatial sharpness and spectral preservation, resulting in spectral and intensity inhomogeneities in the fused image. In this paper, to overcome this problem, we present a robust pansharpening method for multiband images with adaptive spectral–intensity modulation. In this method, we propose an adaptive spectral modulation coefficient (ASMC) and an adaptive intensity modulation coefficient (AIMC) to modulate the spectral and spatial information in the fused image, respectively. Among these coefficients, the ASMC is constructed based on two aspects: first, the details extracted from the panchromatic (PAN) and multispectral (MS) images; and second, the spectral relationship between each MS band. The AIMC is calculated by assessing the correlation and standard deviation between the PAN image and each MS band. Finally, we propose a mathematically linear model to combine ASMC and AIMC to achieve the fused image. Various remote-sensing satellite images were used in the evaluations. Experimental results indicate that the proposed method achieves outstanding performance in balancing spatial and spectral information and outperforms several state-of-the-art fusion methods in terms of both full-reference and no-reference metrics, and on visual inspection.

In the context of automotive driving assistance, this paper describes a generic (i.e. applicable to both vehicle interior and exterior scenes) vision based approach for scene content analysis. It makes use of temporal and spatial information from a stereoscopic sequence of images to localize objects and estimate their position and motion. The proposed method is divided into three steps. First, image features are selected, tracked and reconstructed in the 3D world space. Second, a clustering step is processed in the 5D space made of the positions and 2D motions parameters. The last step is devoted to clusters interpretation: it is out of the scope of the paper, however orientations are given to illustrate the capabilities of the proposed approach. The paper is organized as follows: first, the use of temporal and spatial information from a stereoscopic sequence is investigated. A state of the art of existing methods is presented. Then, a generic approach for object segmentation is proposed. Lastly, experimental results are presented.

In this paper, a soft classification algorithm based on composite kernels, which incorporate both spectral and spatial information, is proposed for hyperspectral image. Compared with hard classification, soft classification provides more information about the probabilities one pixel belongs to each class. To calculate these probabilities, the proposed algorithm uses Support Vector Machine (SVM), and it successfully converts SVM output values into probabilities, while at the same time integrates spatial and spectral information by composite kernels. To validate the proposed algorithm, experiments are conducted on hyperspectral images with 126 and 186 bands, and experimental results show that soft classification using SVM can yield better results compared with Maximum Likelihood Classifier (MLC),and the introduction of spectral-spatial kernels can greatly improve classification accuracies.

The analysis of a multitemporal sequence of images of a given site makes it possible to exploit temporal information in addition to spectral and spatial information, and therefore represents a way to improve the accuracy with respect to the non-contextual single-time classification. The proposed contextual multitemporal classification scheme consists of two stages of multilayer perceptron (MLP) neural networks for each single-time image of the multitemporal sequence. The first stage is a one-hidden layer MLP whose ro/spl circ/le is to estimate, for each pixel, the single-time posterior probability of each class, given the feature vector. These probability estimates represent spectral information; in addition, they are utilized to generate a non-contextual classification map. The neighboring class labels of a given pixel in the non-contextual classification map are exploited to extract spatial information, while temporal information is deduced from the non-contextual maps produced by the remaining single-time images in the multitemporal sequence. Spatial and temporal contextual information together with spectral information serve as inputs for the second stage network of the classification scheme where the fusion takes place. As the network configuration can influence the classification performances, three MLP-based configurations are investigated. Experimental results on a multitemporal data set consisting of two multisensor (Landsat TM and ERS-1 SAR) images are presented. The performances of the proposed methods are discussed and compared with those obtained by a reference classifier based on the Markov random fields fusion approach in terms of classification accuracy. The results show that the proposed fusion approach based on neural networks may represent an interesting solution to the problem of multitemporal contextual fusion.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

Spectral unmixing amounts at estimating the abundance of pure spectral signatures (called endmembers) in each mixed pixel of a hyperspectral image, where mixed pixels arise due to insufficient spatial resolution and other phenomena. A challenging problem is how to automatically identify endmembers, as the presence of mixed pixels generally prevents the localization of pure spectral signatures in transition areas between different land-cover classes. A possible strategy to address this problem is to guide the endmember identification process to spatially homogeneous areas, expected to contain the purest signatures available in the scene. For this purpose, several spatial preprocessing methods have been used prior to endmember identification. However, the preprocessing methods developed thus far only exploit the spatial information and relegate the use of spectral information to the subsequent endmember identification stage. In this paper, we develop a new spatial-spectral preprocessing (SSPP) module which can be used prior to endmember identification and spectral unmixing. The method first derives a spatial homogeneity index for each pixel in the hyperspectral image. This index is relatively insensitive to the noise present in the data. At the same time, it performs unsupervised clustering to identify a set of clusters in spectral space. Finally, it fuses spatial and spectral information by selecting a subset of spatially homogeneous and spectrally pure pixels from each cluster. These pixels constitute the new set of candidates for endmember identification. An innovative contribution of this paper is the combination of spatial and spectral information at the preprocessing stage. Another contribution is the combination, for the first time in the literature, of preprocessing techniques with endmember identification algorithms that do not assume the presence of pure signatures in the scene. An experimental comparison of the proposed method in combination with different endmember identification techniques is conducted using both synthetic and real hyperspectral data collected by the Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS). Our experiments indicate that endmember identification techniques (with and without the pure signature assumption) can greatly benefit from the proposed preprocessing approach, which considers both spatial and spectral information.

We introduce a new spectral mixture analysis approach. Unlike most available approaches that only use the spectral information, this approach uses the spectral and spatial information available in the hyperspectral data. Moreover, it does not assume a global convex geometry model that encompasses all the data but rather multiple local convex models. Both the multiple model boundaries and the model's endmembers and abundances are fuzzy. This allows points to belong to multiple groups with different membership degrees. Our approach is based on minimizing a joint objective function to simultaneously learn the underling fuzzy multiple convex geometry models and find a robust estimate of the model's endmembers and abundances.

A hyperspectral image (HSI) is a collection of several narrow-band images that span a wide spectral range. Each band reflects the same scene, composed of various objects imaged at different wavelengths; the spatial information, however, remains generally consistent across bands. Both types of information, spectral and spatial, can be leveraged to identify and classify objects. Recently, the use of machine learning (ML) in object classification has become increasingly widespread. Regardless of the selected approach, object-specific spectral and spatial information is key to discriminating relevant categories. Whereas spatial information is usually repeated across bands, spectral information tends to be distributed more unevenly and often highly so. This poses the issue of removing redundancy, which is commonly called the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">band selection</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BS</i> ) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">problem</i> and refers to identifying an optimal subset of bands for further HSI processing.