Panchromatic‐Guided Self‐Supervised Framework for Dual‐Camera Compressive Spectral Imaging

Coded Aperture Snapshot Spectral Imaging (CASSI) multiplexes 3D Hyperspectral Images (HSIs) into a 2D sensor to capture dynamic spectral scenes, which, however, sacrifices the spatial information. Dual-Camera Compressive Hyperspectral Imaging (DCCHI) enhances CASSI by incorporating a Panchromatic (PAN) camera to compensate for the loss of spatial information in CASSI. However, the dual-camera structure of DCCHI disrupts the diagonal property of the product of the sensing matrix and its transpose, making it difficult to efficiently and accurately solve the data subproblem in closed-form and thereby hindering the application of model-based methods and Deep Unfolding Networks (DUNs) that rely on such a closed-form solution. To address this issue, we propose an Alternating Direction DUN, named ADRNN, which decouples the imaging model of DCCHI into a CASSI subproblem and a PAN subproblem. The ADRNN alternately solves data terms analytically and a joint prior term in these subproblems. Additionally, we propose a Cross Spectral Transformer (XST) to exploit the joint prior. The XST utilizes cross spectral attention to exploit the correlation between the compressed HSI and the PAN image, and incorporates Grouped-Query Attention (GQA) to alleviate the burden of parameters and computational cost brought by impartially treating the compressed HSI and the PAN image. Furthermore, we built a real DCCHI system and captured large-scale indoor and outdoor scenes for future academic research. Extensive experiments on both simulation and real datasets demonstrate that the proposed method achieves state-of-the-art (SOTA) performance. The code and datasets have been open-sourced at: https://github.com/ShawnDong98/ADRNN-XST.

Coded aperture snapshot spectral imager (CASSI), efficiently captures 3D spectral images by sensing 2D projections of the scene. While CASSI offers a substantial reduction in acquisition time, compared to traditional scanning optical systems, it requires a reconstruction post-processing step. Furthermore, to obtain high-quality reconstructions, an accurate propagation model is required. Notably, CASSI exhibits a variant spatio-spectral sensor response, making it difficult to acquire an accurate propagation model. To address these inherent limitations, this work proposes to learn a deep non-linear fully differentiable propagation model that can be used as a regularizer within an optimization-based reconstruction algorithm. The proposed approach trains the non-linear spatially-variant propagation model using paired compressed measurements and spectral images, by employing side information only in the calibration step. From the deep propagation model incorporation into a plug-and-play alternating direction method of multipliers framework, our proposed method outperforms traditional CASSI linear-based models. Extensive simulations and a testbed implementation validate the efficacy of the proposed methodology.

Coded aperture snapshot spectral imager (CASSI) has been a popular spectral imaging architecture for its ability of capturing hyperspectral images with high temporal resolution. However, such snapshot imaging system entails a large sacrifice in the spatial resolution of the data cube, since only a small amount of light gets into the imager during one snapshot. Also, the spatial resolution of the CASSI system is limited by the pixel size (and amount) of the detector, while it is difficult to fabricate a dense detector with small pixel size, especially for infrared spectral bands. Super-resolution is an advanced post-processing technique to alleviate such problem by exploiting the prior information of the image. In this letter, we try to realize image super-resolution from the perspective of developing new form of measurements by taking advantage of a modified CASSI system equipped with a coded aperture with higher spatial resolution than the detector, merging the SR model into the hardware configuration. Then the original data cube can be reconstructed from lower resolution measurements, thus the super-resolution is realized during the compressive sensing reconstruction process. The new system can be achieved based on the classical CASSI architecture in two dual ways, one by replacing the coded aperture with a higher resolution one and the other by substituting the focal plane array (FPA) detector with a lower resolution one. The experiments show that, we can recover images of higher quality with the first modification of CASSI system above, simply using a higher resolution coded aperture.

A MLP architecture fusing RGB and CASSI for computational spectral imaging

A novel compressive 3D imaging spectrometer based on the coded aperture snapshot spectral imager (CASSI) is proposed. By inserting a microlens array (MLA) into the CASSI system, one can capture spectral data of 3D objects in a single snapshot without requiring 3D scanning. The 3D spatio-spectral sensing phenomena is modelled by computational integral imaging in tandem with compressive coded aperture spectral imaging. A set of focal stack images is reconstructed from a single compressive measurement, and presented as images focused on different depth planes where the objects are located. The proposed optical system is demonstrated with simulations and experimental results.

Coded aperture snapshot spectral imaging (CASSI) reconstructs a hyperspectral image from several two-dimensional (2D) projections via compressive sensing. The reconstruction quality and the sampling efficiency of CASSI can be effectively improved by decreasing the coherence of the underlying sensing matrix. Efforts have been made to minimize the coherence with individual optimization on coded aperture or sparse basis. In this paper, a simultaneous optimization on the system projection and the over-complete dictionary is introduced to minimize the Frobenius norm coherence. The dual-disperser structure and the RGB image sensor are adopted for the lowest coherence in terms of system configuration. The coded aperture and the dictionary are optimized with genetic algorithm and gradient descent respectively, and simultaneous optimization is conducted iteratively. Low coherence of sensing matrix is acquired after the simultaneous optimization, with both reconstruction quality and sampling efficiency significantly improved. Compared to the non-optimized system and state-of-the-art systems with individually optimized coded aperture or dictionary, the simultaneous optimization promotes the peak signal-to-noise ratio by more than 5dB. The coherence minimization via simultaneous optimization on the system matrix and the sparse representation basis may open opportunities for further development of other compressive-sensing-based computational imaging systems.

Coded aperture snapshot spectral imaging (CASSI) has been demonstrated as a feasible solution to recover a 3D hyperspectral image by using a single 2D measurement. In this paper, we propose a new hybrid camera design for CASSI to capture high quality hyperspectral images while maintaining the snapshot advantage. Specifically, we employ a complementary RGB camera in conjunction with the CASSI system. The recorded RGB image can provide reliable spectral clue of the scene. By combining the coded hyperspectral information from the CASSI branch and the uncoded color information from the RGB branch, hyperspectral images can be reconstructed with high fidelity. Furthermore, by conducting demosaicing on the raw RGB image as a preprocessing procedure, even better performance can be achieved. Both theoretical analysis and simulation results show improved accuracy of the proposed method compared to the state-of-the-arts.

Coded aperture snapshot spectral imaging (CASSI) is an effective tool to capture real-world 3D hyperspectral images. While a number of existing work has been conducted for hardware and algorithm design, we make a step towards the low-cost solution that enjoys video-rate high-quality reconstruction. To make solid progress on this challenging yet under-investigated task, we reproduce a stable single disperser (SD) CASSI system to gather large-scale real-world CASSI data and propose a novel deep convolutional network to carry out the real-time reconstruction by using self-attention. In order to jointly capture the self-attention across different dimensions in hyperspectral images (i.e., channel-wise spectral correlation and non-local spatial regions), we propose Spatial-Spectral Self-Attention (TSA) to process each dimension sequentially, yet in an order-independent manner. We employ TSA in an encoder-decoder network, dubbed TSA-Net, to reconstruct the desired 3D cube. Furthermore, we investigate how noise affects the results and propose to add shot noise in model training, which improves the real data results significantly. We hope our large-scale CASSI data serve as a benchmark in future research and our TSA model as a baseline in deep learning based reconstruction algorithms. Our code and data are available at https://github.com/mengziyi64/TSA-Net .

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

Coded aperture snapshot spectral imager (CASSI) based on compressed sensing can capture a three-dimensional (3-D) data cube information in one snapshot without scanning, so it is a kind of novel spectral imaging technique with good prospects. In this paper, the data flow procession of two kind of CASSI were studied, which similarities and differences were depicted. The basic principle of the encoded representation of both spectral information and 2-D spatial information of a scene were presented. The numerical estimation techniques based on the compressive sensing theory were applied to reconstruct 3-D data cube.

The coded aperture snapshot spectral imaging (CASSI) architecture has been employed widely for capturing hyperspectral video. Despite allowing concurrent capture of hyperspectral video, spatial modulation in CASSI sacrifices image resolution significantly while reconstructing spectral projection via sparse sampling. Several multiview alternatives have been proposed to handle this low spatial resolution problem and improve measurement accuracy, for instance, by adding a translation stage for the coded aperture or changing the static coded aperture with a digital micromirror device for dynamic modulation. State‐of‐the‐art solutions enhance spatial resolution significantly but are incapable of capturing video using CASSI. In this paper, we present a novel compressive coded aperture imaging design that increases spatial resolution while capturing 4D hyperspectral video of dynamic scenes. We revise the traditional CASSI design to allow for multiple sampling of the randomness of spatial modulation in a single frame. We demonstrate that our compressive video spectroscopy approach yields enhanced spatial resolution and consistent measurements, compared with the traditional CASSI design.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

In the coded aperture snapshot spectral imaging (CASSI) system, the coded and compressed single-channel measurements need to be reconstructed into hyperspectral cubes. Existing discriminative models reconstruct the spectral cube by optimizing the mean squared error (MSE) between the ground truth and the predicted image, employing peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as metrics to gauge the quality of reconstruction. However, these indicators often possess significant limitations in mimicking human visual perception and in discerning the impact of image distortions on perceived visual quality. In this article, a new model named CASSIDiff is proposed to reconstruct CASSI measurements, achieving advanced results in perceptual loss-based evaluation metrics such as learned perceptual image patch similarity (LPIPS) and Fréchet inception distance (FID). The diffusion model, which enjoys high accuracy and reliability in generative tasks, is used for the first time for the hyperspectral reconstruction task. A feature fusion mechanism based on discrete wavelet transform (DWT) is used to weaken the noise interference effect in the conditional diffusion model. Considering the interspectra similarity and long-range dependencies of hyperspectral data, the spatial-spectral attention mechanism is also introduced. Experiments show that CASSIDiff not only outperforms most existing algorithms in simulation datasets but also shows robustness to real data published and collected in our home-built CASSI system. The code and models are publicly available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YifanSi/CASSIDiff</uri>.

Coded Aperture Snapshot Spectral Imaging (CASSI) is an effective tool to capture spectral images, which has the advantages of snapshot imaging, high luminous flux, high signal-to-noise ratio and low sampling frequency. However, conventional CASSI generally uses refractive prisms or gratings for spectral dispersion, which leads to the nonlinear dispersion phenomenon and the requirement of large detector chip respectively. To overcome these issues, conventional refractive prisms or gratings are replaced by an axially dispersive diffractive optical element (DOE, i.e., computational optics) together with a RGB Bayer filter (i.e., a color-coded aperture) in this study. Specifically, the spatial-spectral information of a test scene is jointly modulated by the DOE and the Bayer filter integrated with a sensor chip. A fully differentiable imaging model is built based on the principle of diffractive optics and the deep learning technology. Furthermore, an optimization design of the DOE with the coded aperture is realized through an end-to-end approach, the output spectral images of which are restored by a Res-Unet neural network. Several simulation results show that up to 31 high-fidelity spectral bands in the range of 400 to 700 nm with a good spatial and spectral resolution can be recovered by the proposed snapshot system.

Multi-shot coded aperture snapshot spectral imaging (CASSI) uses multiple measurement snapshots to encode the three-dimensional hyperspectral image (HSI). Increasing the number of snapshots will multiply the number of measurements, making CASSI system more appropriate for detailed spatial or spectrally rich scenes. However, the reconstruction algorithms still face the challenge of being ineffective or inflexible. In this paper, we propose a plug-and-play (PnP) method that uses denoiser as priors for multi-shot CASSI. Specifically, the proposed PnP method is based on the primal-dual algorithm with linesearch (PDAL), which makes it flexible and can be used for any multi-shot CASSI mechanisms. Furthermore, a new subspaced-based nonlocal reweighted low-rank (SNRL) denoiser is presented to utilize the global spectral correlation and nonlocal self-similarity priors of HSI. By integrating the SNRL denoiser into PnP-PDAL, we show the balloons (512 × 512 × 31) in CAVE dataset recovered from two snapshots compressive measurements with MPSNR above 50 dB. Experimental results demonstrate that our proposed method leads to significant improvements compared to the current state-of-the-art methods.