Mamba-based multiview spatial and frequency collaborative network for hyperspectral image unmixing

Hyperspectral image unmixing is an important part of hyperspectral data analysis. The mixed pixel decomposition consists of two steps, endmember (the unique signatures of pure ground components) extraction and abundance (the proportion of each endmember in each pixel) estimation. Recently, a Discrete Particle Swarm Optimization algorithm (DPSO) was proposed for accurately extract endmembers with high optimal performance. However, the DPSO algorithm shows very high computational complexity, which makes the endmember extraction procedure very time consuming for hyperspectral image unmixing. Thus, in this paper, the DPSO endmember extraction algorithm was parallelized, implemented on the CUDA (GPU K20) platform, and evaluated by real hyperspectral remote sensing data. The experimental results show that with increasing the number of particles the parallelized version obtained much higher computing efficiency while maintain the same endmember exaction accuracy.

Hyperspectral unmixing aims to estimate endmember spectra and their corresponding abundance fractions at the subpixel scale, which is a critical preprocessing step for quantitative analysis of hyperspectral remote sensing imagery. While deep learning-based methods have achieved remarkable progress, three fundamental challenges remain: (i) reliance on a single shared spatial prior that cannot decouple the heterogeneous spatial patterns of different land covers; (ii) the lack of synergy in jointly optimizing endmember extraction and abundance estimation; (iii) the poor robustness of unsupervised training to complex mixtures, noise, and class imbalance. To address these issues, we propose a novel unsupervised unmixing framework that integrates adaptive orthogonal multi-faceted graph representation with curriculum learning. Specifically, we design an Adaptive Orthogonal Multi-Faceted Graph Generator (AOMFG) to learn a set of independent orthogonal graph structures, achieving spatially informed decoupling of land cover patterns. Then, a dual-branch collaborative optimization network is constructed: a Graph Convolutional Network (GCN) branch that incorporates the learned spatial topological priors for abundance estimation, and a 1D Convolutional Neural Network (1DCNN) branch that employs a query-attention mechanism to adaptively aggregate pure spectral features for endmember extraction. Finally, we introduce a three-stage curriculum learning strategy that progressively fine-tunes the model, which significantly enhances its performance. Extensive experiments on three widely used real-world benchmark datasets demonstrate that our proposed framework consistently outperforms state-of-the-art methods in both endmember extraction and abundance estimation accuracy. Comprehensive ablation studies, parameter sensitivity analysis, and noise robustness tests further validate the effectiveness of each core component.

Deep learning-based hyperspectral unmixing (HU) is getting increasing attention in the field of remote sensing, aiming at endmember extraction and abundance estimation at pixel scale. However, many existing deep learning-based unmixing methods base on linear mixing models, neglecting complex nonlinear light scattering interactions. Furthermore, these methods often treat all spectral bands indiscriminately, ignoring characteristic differences between endmembers, hampering endmember separation. To address these issues, we present BU-Net, a novel approach for HU based on the generalized bilinear mixing model (GBM), which is a two-stream stacked autoencoder architecture designed to enhance inter-class separability. In the encoder, we employ 3D convolutions with multiple receptive field to extract multiscale spatial and spectral features simultaneously. Additionally, we design a novel band selection based on inter-class separability (BSICS), which identifies bands with inter-class separability (BICS) and the obtained bands are taken as an additional stream for improving performance. In the decoder, BU-Net develops a two-stream structure encompassing linear and bilinear elements, aligning with the theoretical components and constraints of GBM. To further enhance separability between endmembers during training, we use the spectral angle distance between BICS and its reconstruction as a loss regularization term. Moreover, we utilize materials' representative pixels obtained in the process of BSICS to initialize endmembers, which offers effective guidance for modeling the spectral properties. Experimental results on synthetic and real hyperspectral datasets show that our method outperforms state-of-the-art methods. This novel approach addresses limitations of linear mixing models while leveraging deep learning to improve accuracy of HU.

Previous studies on Sound Event Localization and Detection (SELD) have primarily focused on CNN- and Transformer-based designs. While CNNs possess local receptive fields, making it difficult to capture global dependencies over long sequences, Transformers excel at modeling long-range dependencies but have limited sensitivity to local time–frequency features. Recently, the VMamba architecture, built upon the Visual State Space (VSS) model, has shown great promise in handling long sequences, yet it remains limited in modeling local spatial details. To address this issue, we propose a novel state space model with an attention-enhanced feature fusion mechanism, termed FFMamba, which balances both local spatial modeling and long-range dependency capture. At a fine-grained level, we design two key modules: the Multi-Scale Fusion Visual State Space (MSFVSS) module and the Wavelet Transform-Enhanced Downsampling (WTED) module. Specifically, the MSFVSS module integrates a Multi-Scale Fusion (MSF) component into the VSS framework, enhancing its ability to capture both long-range temporal dependencies and detailed local spatial information. Meanwhile, the WTED module employs a dual-branch design to fuse spatial and frequency domain features, improving the richness of feature representations. Comparative experiments were conducted on the DCASE2021 Task 3 and DCASE2022 Task 3 datasets. The results demonstrate that the proposed FFMamba model outperforms recent approaches in capturing long-range temporal dependencies and effectively integrating multi-scale audio features. In addition, ablation studies confirmed the effectiveness of the MSFVSS and WTED modules.

The spatial resolution of hyperspectral data is low, so there are a large number of mixed pixels, which is also one of the main reasons that reduce the accuracy of hyperspectral image target classification. Hyperspectral unmixing is an important subject in the field of remote sensing. Hyperspectral unmixing generally consists of three steps: reduction, endmember extraction and inversion. As one of the key steps of hyperspectral unmixing, efficient and rapid endmember extraction is an important object in hyperspectral remote sensing. In this paper, the endmember extraction of hyperspectral data is implemented based on PCA and a new SGA algorithm, which solves the dimension limitation of traditional SGA algorithm and the new SGA algorithm without data redundancy caused by data dimensionality reduction.

Endmember extraction is a key step in hyperspectral unmixing. This paper proposes a new endmember extraction framework based on the swarm intelligence algorithm. We adopt a discrete structure because pixels exist within a discrete frame. Traditional swarm intelligence algorithms produce stacked solutions based on similar endmembers in the same class. We introduce a “distance” factor into the objective function to limit the number of endmembers per class. We then propose three endmember extraction methods based on the artificial bee colony (ABC), ant colony optimization (ACO) and particle swarm optimization (PSO) algorithms. Experiments with both simulated and actual hyperspectral image data indicate that the proposed framework can significantly improve the accuracy of endmember extraction.

Hyperspectral Unmixing (HU), also known as spectral mixture analysis, is a challenging problem that decomposes a mixed spectrum into a collection of endmembers and their abundance fractions. In this paper, we extended the autoencoder network (Palsson et al. in IEEE Access 6:25646–25656, [1]) for blind hyperspectral nonlinear unmixing. The proposed autoencoder architecture consists of two networks encoder and decoder. The encoder network has the same as the original architecture. The architecture of the decoder network was altered to handle nonlinear unmixing. Our proposed encoder network has four fully connected layers, each with the number of neurons equal to the dimension of the end members. Experiments were conducted using nonlinear synthetic dataset sampled from the USGS library, and the performance was evaluated using both SAD and SID metrics. Four categories of experiments were carried out; accuracy assessment, weight initialization techniques, learning rate, and robustness to the noise. Experimental results show that the proposed autoencoder outperforms traditional endmember extraction algorithms in nonlinear cases. Finally, we introduced the application of hyperspectral image unmixing algorithm in the Internet of things (IoT) environment.

Endmember extraction (EE) plays a crucial part in the hyperspectral unmixing (HU) process. To obtain satisfactory EE results, the EE can be considered as the multiobjective optimization problem to optimize the volume maximization (VM) and root-mean-square error (RMSE) simultaneously. However, it is often quite challenging to balance the conflict of these objectives. In order to tackle the challenges of multiobjective EE, we present a (μ + λ) multiobjective differential evolution algorithm ((μ + λ)-MODE) based on ranking multiple mutations. In the (μ + λ)-MODE algorithm, ranking multiple mutations are adopted to create the mutant vectors via the scaling factor pool to enhance the population diversity. Moreover, mutant vectors employ the binary crossover operator to generate the trial vectors through a crossover control parameter pool in (μ + λ)-MODE to take advantage of the good information of the population. In addition, (μ + λ)-MODE utilizes the fast nondominated sorting approach to sort the parent and trial vectors, and then selects the elitism offspring as the next population via the (μ + λ) selection strategy. Eventually, experimental comparative results in three real HSIs reveal that our proposed (μ + λ)-MODE is superior to other EE methods.

Understanding the neural representation of spatial frequency (SF) in the primate cortex is vital for unraveling visual processing in object recognition. While many studies focus on SF in the primary visual cortex, the characteristics and interaction of SF with category representation remain unclear. To explore SF representation in the inferior temporal (IT) cortex of macaques, we conducted extracellular recordings with complex stimuli systematically filtered by SF. Our findings reveal explicit SF coding at both single-neuron and population levels. Temporal dynamics analysis of SF representation reveals that low SF (LSF) is decoded faster than high SF (HSF), and the SF preference dynamically shifts from LSF to HSF over time. Additionally, the SF representation for each neuron forms a profile that predicts category selectivity at the population level. IT neurons cluster into four groups based on SF preference, each with distinct category coding behaviors. Notably, HSF-preferring neurons show the highest category decoding for faces. Despite the existing connection between SF and category coding, we have identified uncorrelated representations of SF and category. Unlike category coding, SF is more sparsely represented and depends more on individual neurons. These findings dissociate SF and category representations, underscoring SF's pivotal role in object recognition.

Hyperspectral unmixing (HU) has become an important technique in exploiting hyperspectral data since it decomposes a mixed pixel into a collection of endmembers weighted by fractional abundances. The endmembers of a hyperspectral image (HSI) are more likely to be generated by independent sources and be mixed in a macroscopic degree before arriving at the sensor element of the imaging spectrometer as mixed spectra. Over the past few decades, many attempts have focused on imposing auxiliary regularizes on the conventional nonnegative matrix factorization (NMF) framework in order to effectively unmix these mixed spectra. As a promising step toward finding an optimum regularizer to extract endmembers, this paper presents a novel blind HU algorithm, referred to as Kurtosis-based Smooth Nonnegative Matrix Factorization (KbSNMF) which incorporates a novel regularizer based on the statistical independence of the probability density functions of endmember spectra. Imposing this regularizer on the conventional NMF framework promotes the extraction of independent endmembers while further enhancing the parts-based representation of data. Experiments conducted on diverse synthetic HSI datasets (with numerous numbers of endmembers, spectral bands, pixels, and noise levels) and three standard real HSI datasets demonstrate the validity of the proposed KbSNMF algorithm compared to several stateof- the-art NMF-based HU baselines. The proposed algorithm exhibits superior performance especially in terms of extracting endmember spectra from hyperspectral data; therefore, it could uplift the performance of recent deep learning HU methods which utilize the endmember spectra as supervisory input data for abundance extraction.

The normal compositional model (NCM) is a well-known and powerful model in hyperspectral unmixing which represents endmembers as independent Gaussian vectors to capture endmember variability. However, the assumption of independent endmembers diminishes the model accuracy because the high degree of correlation between endmembers of a scene and identical sources of variability demonstrate that the endmembers are dependent. This paper proposes a new hyperspectral unmixing algorithm which represents endmembers using dependent Gaussian vectors to estimate abundance fractions. To overcome the higher complexity caused by dependence assumption, this algorithm introduces new independent Gaussian vectors named Base Vectors to represent different endmembers by a weighted linear combination. Also, the proposed unmixing algorithm uses maximum likelihood method to estimate weight coefficients of Base Vectors which are used to represent mixed pixel. Finally, abundance estimation can be done using the new representation for endmembers and mixed pixel. The proposed algorithm is evaluated and compared with other state-of-the-art unmixing algorithms using simulated and real hyperspectral images. Experimental results demonstrate that the proposed unmixing algorithm can unmix pixels composed of correlated endmembers in hyperspectral images in the presence of spectral variability more accurately than previous methods.

Spectral unmixing is an important problem for remotely sensed hyperspectral data exploitation. Automatic spectral unmixing can be viewed as a three-stage problem, where the first stage is subspace identification, the next one is endmember extraction, and the final one is abundance estimation. In this sequence, endmember extraction is the most challenging problem. Many researchers have attempted to extract endmembers from hyperspectral images using spectral information only. However, it is well known that the inclusion of spatial information can improve the endmember extraction task. In this article, we introduce a new endmember extraction algorithm that exploits both spectral and spatial information. A main innovation of the proposed algorithm is that spatial information is exploited using entropy, while spectral information is exploited using convex set optimization. In the literature, none of the spatial–spectral algorithms has used entropy as spatial information. The inclusion of this entropy-based spatial information improves the accuracy of the endmember extraction process. The results obtained by the proposed algorithm are compared (using a variety of metrics) with those obtained by other state-of-the-art methods, using both synthetic and real datasets. Our experimental results demonstrate that the proposed algorithm outperforms many available algorithms.

Few-shot hyperspectral classification is a challenging problem that involves obtaining effective spatial–spectral features in an unsupervised or semi-supervised manner. In recent years, as a result of the development of computer vision, deep learning techniques have demonstrated their superiority in tackling the problems of hyperspectral unmixing (HU) and classification. In this paper, we present a new semi-supervised pipeline for few-shot hyperspectral classification, where endmember abundance maps obtained by HU are treated as latent features for classification. A cube-based attention 3D convolutional autoencoder network (CACAE), is applied to extract spectral–spatial features. In addition, an attention approach is used to improve the accuracy of abundance estimation by extracting the diagnostic spectral features associated with the given endmember more effectively. The endmember abundance estimated by the proposed model outperforms other convolutional neural networks (CNNs) with respect to the root mean square error (RMSE) and abundance spectral angle distance (ASAD). Classification experiments are performed on real hyperspectral datasets and compared to several supervised and semi-supervised models. The experimental findings demonstrate that the proposed approach has promising potential for hyperspectral feature extraction and has better performance relative to CNN-based supervised classification under small-sample conditions.

Recently, deep learning has been widely applied in the field of blind hyperspectral unmixing (HU), which aims to simultaneously estimate constitutive endmembers and their abundances in hyperspectral images (HSIs). Generally, the HU process based on deep-learning methods consists of two parts: an encoder and a decoder. In many networks, the decoder stage uses the extracted semantic information of the HSI by the encoder, without direct access to the manifold structure of the HSI. To address this limitation and simultaneously capture both the semantic information and manifold structure of the HSI, in this letter, we propose a dual-channel enhanced decoder network (DED-Net) for the HU problem. Specifically, DED-Net redesigns a decoder by adding a dual-channel graph regularizer that establishes a physically meaningful immediate connection between the abundance and the HSI, effectively integrating both the information from the encoder and the original HSI to enhance endmembers and abundance estimation. Experimental results demonstrate the superiority of our proposed method, which leads to a more accurate unmixing performance.

Reconstructing degraded images is a critical task in image processing. Although CNN and Transformer-based models are prevalent in this field, they exhibit inherent limitations, such as inadequate long-range dependency modeling and high computational costs. To overcome these issues, we introduce the Channel-Aware U-Shaped Mamba (CU-Mamba) model, which incorporates a dual State Space Model (SSM) framework into the U-Net architecture. Build on top of hierarchical structure and attention mechanism, CU-Mamba employs a Spatial SSM module for global context encoding and a Channel SSM component to preserve channel correlation features, both in linear computational complexity relative to the feature map size. Extensive experimental results validate CU-Mamba's superiority over existing state-of-the-art methods, underscoring the importance of integrating both spatial and channel contexts in image restoration.