Coded Aperture Design for Compressive Spectral Subspace Clustering

LADMM-Net: An unrolled deep network for spectral image fusion from compressive data

Compressive spectral imaging (CSI) allows the acquisition of the spectral information of a three-dimensional scenes by using coded projections in a sensor with lower dimension. However, the compressed sampling of information with simultaneously high spatial and high spectral resolution demands expensive high-resolution sensors. One of the main challenges in CSI is to obtain a high-quality image of high-resolution reconstructions using low-cost architectures. Single pixel camera is an approach that has had a high impact in spectroscopy, due to its low-cost implementation compared to CSI architectures with 2D sensors. On the other hand, recent works have been shown that image fusion using measurements from a CSI sensor based on side information leads to improvement in the quality of the fused image. This work proposes a spectral image fusion methodology for increasing the spatio-spectral resolution through side information and at the same time improve the reconstruction quality of the data cube with a low-cost architecture, optimizing the similarity of the reconstructed spectral image with each sensor. Simulations and experimental results for the proposed methodology show that improve the quality of the reconstruction in up to 11 dB with respect to the traditional approach of upsampling the single pixel image reconstruction through bilinear interpolation.

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

Compressive spectral imaging (CSI) acquires coded projections of a spectral image by performing a modulation of the data cube followed by a spectral-wise integration. To avoid the spectral image reconstruction procedure, this paper proposes a classification approach that extracts features directly from multi-sensor CSI measurements. Particularly, the proposed method obtains the features by considering the spectral information extracted from Hyperspectral CSI measurements, and the local spatial information extracted by clustering the Multispectral CSI measurements using a superpixel algorithm. This approach is evaluated on Pavia University and Salinas Valley datasets. Extensive simulations show that considering the local spatial information boosts the overall accuracy up to 3% in comparison with traditional approaches that only uses the spectral information. Furthermore, the computation time of the approach that reconstructs, fuses and classifies takes approximately 87.43 [s], while classifying directly from multi-sensor compressive measurements takes only 0.74 [s], achieving similar classification results.

Compressive spectral imaging (CSI) enables the acquisition of spectral and spatial information of a scene using fewer projected measurements than traditional scanning approaches. Recently, research efforts have focused on obtaining high-resolution spectral images via expensive detectors and sophisticated CSI devices. Alternatively, high-resolution spectral images can be obtained using side information or fusion of compressed measurements, without significantly increasing acquisition costs. Indeed, these approaches retrieve improved resolution images applying iterative and computationally expensive algorithms. This paper proposes the fusion of compressed measurements obtained from two state-of-the-art CSI systems, the single-pixel camera (SPC) and the three-dimensional coded aperture snapshot imaging system (3D-CASSI), such that high-resolution images can be obtained by exploiting detailed spectra provided by the SPC and high spatial resolution of the 3D-CASSI. Specifically, a noniterative reconstruction algorithm is proposed, based on the fact that the spatial-spectral data lie in a low-dimensional subspace. In contrast to related works, the proposed approach relies on implementable CSI systems. Simulations and experimental results show the effectiveness of the proposed method compared to similar approaches, both in reconstruction quality and complexity. Specifically, the proposed method is up to 5.6 times faster than its counterparts and provides comparable quality of attained reconstructions.

Compressive spectral imaging has garnered significant attention for its ability to effectively enhance the captured spatial and spectral information. Predominant methods, based on compressive sensing, typically formulate the imaging task as a constrained optimization problem and rely on hand-crafted priors to model the sparsity of spectral images. However, these approaches often suffer from suboptimal performance due to the inherent difficulty of identifying an appropriate transform space where spectral images exhibit sparsity. To overcome this limitation, we propose a novel convolutional sparse coding-inspired untrained network prior for fast and adaptive identification of the sparse transform domain and compressible signal. Specifically, a Lightweight Convolutional Thresholding sparse Coding (LCTC) network is designed as the sparse transform domain, with its inputs interpreted as sparse coefficients. Crucially, both the transform domain and its coefficients are solved in a self-supervised learning manner. Furthermore, we demonstrate that LCTC prior can be seamlessly incorporated into the iterative optimization algorithm as a Plug-and-Play (PnP) regularization. Both the LCTC and PnP-LCTC exhibit superior performance compared to previous methods. Experiments under various scenarios validate the effectiveness and efficiency of our approach.

DMP-Net: Deep semantic prior compressed spectral reconstruction method towards intraoperative imaging of brain tissue.

Massive amounts of data in spectral imagery increase acquisition, storing and processing costs. Compressive spectral imaging (CSI) methods allow the reconstruction of spatial and spectral information from a small set of random projections. The single pixel camera is a low cost optical architecture which enables the compressive acquisition of spectral images. Traditional CSI reconstruction methods obtain a sparse approximation of the underlying spatial and spectral information, however the complexity of these algorithms increases in proportion to the dimensionality of the data. This work proposes a multiresolution (MR) CSI reconstruction approach from single pixel camera measurements that exploits spectral similarities between pixels to group them in super-pixels such that the total number of unknowns in the inverse problem is reduced. Specifically, two different types of super-pixels are considered: rectangular and irregular structures. Simulation and experimental results show that the proposed MR scheme improves reconstruction quality in up to 6dB of PSNR and reconstruction time in up to 90% with respect to the traditional full resolution reconstructions.

Subspace clustering methods with sparsity prior, such as Sparse Subspace Clustering (SSC) [1], are effective in partitioning the data that lie in a union of subspaces. Most of those methods require certain assumptions, e.g. independence or disjointness, on the subspaces. These assumptions are not guaranteed to hold in practice and they limit the application of existing sparse subspace clustering methods. In this paper, we propose \(\ell ^{0}\)-induced sparse subspace clustering (\(\ell ^{0}\)-SSC). In contrast to the required assumptions, such as independence or disjointness, on subspaces for most existing sparse subspace clustering methods, we prove that subspace-sparse representation, a key element in subspace clustering, can be obtained by \(\ell ^{0}\)-SSC for arbitrary distinct underlying subspaces almost surely under the mild i.i.d. assumption on the data generation. We also present the “no free lunch” theorem that obtaining the subspace representation under our general assumptions can not be much computationally cheaper than solving the corresponding \(\ell ^{0}\) problem of \(\ell ^{0}\)-SSC. We develop a novel approximate algorithm named Approximate \(\ell ^{0}\)-SSC (\(\hbox {A}\ell ^{0}\)-SSC) that employs proximal gradient descent to obtain a sub-optimal solution to the optimization problem of \(\ell ^{0}\)-SSC with theoretical guarantee, and the sub-optimal solution is used to build a sparse similarity matrix for clustering. Extensive experimental results on various data sets demonstrate the superiority of \(\hbox {A}\ell ^{0}\)-SSC compared to other competing clustering methods.

Long-wave infrared (LWIR) spectral imaging plays a critical role in various applications such as gas monitoring, mineral exploration, and fire detection. Recent advancements in computational spectral imaging, powered by advanced algorithms, have enabled the acquisition of high-quality spectral images in real time, such as with the Uncooled Snapshot Infrared Spectrometer (USIRS). However, the USIRS system faces challenges, particularly a low spectral resolution and large amount of data noise, which can degrade the image quality. Deep learning has emerged as a promising solution to these challenges, as it is particularly effective at handling noisy data and has demonstrated significant success in hyperspectral imaging tasks. Nevertheless, the application of deep learning in LWIR imaging is hindered by the severe scarcity of long-wave hyperspectral image data, which limits the training of robust models. Moreover, existing networks that rely on convolutional layers or attention mechanisms struggle to effectively capture both local and global spectral correlations. To address these limitations, we propose the pixel-based Hierarchical Spectral Transformer (HST), a novel deep learning architecture that learns from publicly available single-pixel long-wave infrared spectral databases. The HST is designed to achieve a high spectral resolution for LWIR spectral image reconstruction, enhancing both the local and global contextual understanding of the spectral data. We evaluated the performance of the proposed method on both simulated and real-world LWIR data, demonstrating the robustness and effectiveness of the HST in improving the spectral resolution and mitigating noise, even with limited data.

The main contribution of the paper is a new approach to subspace clustering that is significantly more computationally efficient and scalable than existing state-of-the-art methods. The central idea is to modify the regression technique in sparse subspace clustering (SSC) by replacing the $\ell_{1}$ minimization with a generalization called Ordered Weighted $\ell_{1}$ (OWL) minimization which performs simultaneous regression and clustering of correlated variables. Using random geometric graph theory, we prove that OWL regression selects more points within each subspace than $\ell_{1}$, resulting in better clustering results. This allows for accurate subspace clustering based on regression solutions for only a small subset of the total dataset, significantly reducing the computational complexity compared to SSC. In experiments, we find that our OWL approach can achieve a speedup of $ 20\times$ to $30\times$ for synthetic problems and $4\times$ to $8\times$ on real data problems.

Compressive Covariance Sampling (CCS) is a strategy used to recover the covariance matrix (CM) directly from compressive measurements. Several works have proven the advantages of CSS in Compressive Spectral Imaging (CSI) but most of these algorithms require multiple random projections of the scene to obtain good reconstructions. However, several low-resolution copies of the scene can be captured in a single snapshot through a lenslet array. For this reason, this paper proposes a sensing protocol and a single snapshot CCS optical architecture using a lenslet array based on the Dual Dispersive Aperture Spectral Imager(DD-CASSI) that allows the recovery of the covariance matrix with a single snapshot. In this architecture uses the lenslet array allows to obtain different projections of the image in a shot due to the special coded aperture. In order to validate the proposed approach, simulations evaluated the quality of the recovered CM and the performance recovering the spectral signatures against traditional methods. Results show that the image reconstructions using CM have PSNR values about 30 dB, and reconstructed spectrum has a spectral angle mapper (SAM) error less than 15° compared to the original spectral signatures.

Snapshot Compressive Spectral Imaging Systems (SCSI) compress the scenes by capturing 2D projections of the encoded underlying signals. A decoder, trained with pre-acquired datasets, reconstructs the spectral images. SCSI systems based on diffractive optical elements (DOE) provide a small form factor and the single DOE can be optimized in an end-to-end manner. Since the spectral image is highly compressed in a SCSI system based on a single DOE, the quality of image reconstruction can be insufficient for diverse spectral imaging applications. This work proposes a multishot spectral imaging system employing a double-phase encoding with a double DOE architecture (DoDo), to improve the spectral reconstruction performance. The first DOE is fixed and provides the benefits of the diffractive optical systems. The second DOE provides the variable encoding of the multishot architectures. The work presents a differentiable mathematical model for the multishot DoDo system and optimizes the parameters of the DoDo architecture in an end-to-end manner. The proposed system was tested using simulations and a hardware prototype. To obtain a low-cost system, the implementation uses a deformable mirror for the variable DOE. The proposed DoDo system shows an improvement of up to 4dB in PSNR in the reconstructed spectral images compared with the single DOE system.

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

We propose Ordered Subspace Clustering (OSC) to segment data drawn from a sequentially ordered union of subspaces. Current subspace clustering techniques learn the relationships within a set of data and then use a separate clustering algorithm such as NCut for final segmentation. In contrast our technique, under certain conditions, is capable of segmenting clusters intrinsically without providing the number of clusters as a parameter. Similar to Sparse Subspace Clustering (SSC) we formulate the problem as one of finding a sparse representation but include a new penalty term to take care of sequential data. We test our method on data drawn from infrared hyper spectral data, video sequences and face images. Our experiments show that our method, OSC, outperforms the state of the art methods: Spatial Subspace Clustering (SpatSC), Low-Rank Representation (LRR) and SSC.