Deep Learned Non-Linear Propagation Model Regularizer for Compressive Spectral Imaging

A MLP architecture fusing RGB and CASSI for computational spectral imaging

Coded aperture snapshot spectral imager (CASSI) has been a popular spectral imaging architecture for its ability of capturing hyperspectral images with high temporal resolution. However, such snapshot imaging system entails a large sacrifice in the spatial resolution of the data cube, since only a small amount of light gets into the imager during one snapshot. Also, the spatial resolution of the CASSI system is limited by the pixel size (and amount) of the detector, while it is difficult to fabricate a dense detector with small pixel size, especially for infrared spectral bands. Super-resolution is an advanced post-processing technique to alleviate such problem by exploiting the prior information of the image. In this letter, we try to realize image super-resolution from the perspective of developing new form of measurements by taking advantage of a modified CASSI system equipped with a coded aperture with higher spatial resolution than the detector, merging the SR model into the hardware configuration. Then the original data cube can be reconstructed from lower resolution measurements, thus the super-resolution is realized during the compressive sensing reconstruction process. The new system can be achieved based on the classical CASSI architecture in two dual ways, one by replacing the coded aperture with a higher resolution one and the other by substituting the focal plane array (FPA) detector with a lower resolution one. The experiments show that, we can recover images of higher quality with the first modification of CASSI system above, simply using a higher resolution coded aperture.

Coded aperture snapshot spectral imager (CASSI) based on compressed sensing can capture a three-dimensional (3-D) data cube information in one snapshot without scanning, so it is a kind of novel spectral imaging technique with good prospects. In this paper, the data flow procession of two kind of CASSI were studied, which similarities and differences were depicted. The basic principle of the encoded representation of both spectral information and 2-D spatial information of a scene were presented. The numerical estimation techniques based on the compressive sensing theory were applied to reconstruct 3-D data cube.

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

Coded Aperture Snapshot Spectral Imaging (CASSI) is an effective tool to capture spectral images, which has the advantages of snapshot imaging, high luminous flux, high signal-to-noise ratio and low sampling frequency. However, conventional CASSI generally uses refractive prisms or gratings for spectral dispersion, which leads to the nonlinear dispersion phenomenon and the requirement of large detector chip respectively. To overcome these issues, conventional refractive prisms or gratings are replaced by an axially dispersive diffractive optical element (DOE, i.e., computational optics) together with a RGB Bayer filter (i.e., a color-coded aperture) in this study. Specifically, the spatial-spectral information of a test scene is jointly modulated by the DOE and the Bayer filter integrated with a sensor chip. A fully differentiable imaging model is built based on the principle of diffractive optics and the deep learning technology. Furthermore, an optimization design of the DOE with the coded aperture is realized through an end-to-end approach, the output spectral images of which are restored by a Res-Unet neural network. Several simulation results show that up to 31 high-fidelity spectral bands in the range of 400 to 700 nm with a good spatial and spectral resolution can be recovered by the proposed snapshot system.

A novel compressive 3D imaging spectrometer based on the coded aperture snapshot spectral imager (CASSI) is proposed. By inserting a microlens array (MLA) into the CASSI system, one can capture spectral data of 3D objects in a single snapshot without requiring 3D scanning. The 3D spatio-spectral sensing phenomena is modelled by computational integral imaging in tandem with compressive coded aperture spectral imaging. A set of focal stack images is reconstructed from a single compressive measurement, and presented as images focused on different depth planes where the objects are located. The proposed optical system is demonstrated with simulations and experimental results.

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

Imaging spectroscopy involves the sensing of a large amount of spatial information across a multitude of wavelengths. Conventional approaches to hyperspectral sensing scan adjacent zones of the underlying spectral scene and merge the results to construct a spectral data cube. Push broom spectral imaging sensors, for instance, capture a spectral cube with one focal plane array (FPA) measurement per spatial line of the scene [1], [2]. Spectrometers based on optical bandpass filters sequentially scan the scene by tuning the bandpass filters in steps. The disadvantage of these techniques is that they require scanning a number of zones linearly in proportion to the desired spatial and spectral resolution. This article surveys compressive coded aperture spectral imagers, also known as coded aperture snapshot spectral imagers (CASSI) [1], [3], [4], which naturally embody the principles of compressive sensing (CS) [5], [6]. The remarkable advantage of CASSI is that the entire data cube is sensed with just a few FPA measurements and, in some cases, with as little as a single FPA shot.

This work describes numerical methods for the joint reconstruction and segmentation of spectral images taken by compressive sensing coded aperture snapshot spectral imagers (CASSI). In a snapshot, a CASSI captures a two-dimensional (2D) array of measurements that is an encoded representation of both spectral information and 2D spatial information of a scene, resulting in significant savings in acquisition time and data storage. The reconstruction process decodes the 2D measurements to render a three-dimensional spatio-spectral estimate of the scene and is therefore an indispensable component of the spectral imager. In this study, we seek a particular form of the compressed sensing solution that assumes spectrally homogeneous segments in the two spatial dimensions, and greatly reduces the number of unknowns, often turning the underdetermined reconstruction problem into one that is overdetermined. Numerical tests are reported on both simulated and real data representing compressed measurements.

Conventional multispectral imaging systems based on bandpass filters struggle to record multispectral videos with high spatial resolutions because of their limited light efficiencies. This paper proposes a multi-aperture multispectral imaging system based on notch filters that overcomes this limitation by allowing light from most of the spectrum to pass through. Based on this imaging principle, a prototype multi-aperture multispectral imaging system comprising notch filters was built and demonstrated. Further, a dictionary learning- and total variation-based spectral super-resolution algorithm was developed to reconstruct spectral images. The simulation results obtained using public multispectral datasets showed that, compared to the dictionary learning-based spectral super-resolution algorithm, the proposed algorithm reconstructed the spectral information with a higher accuracy and removed noise, and the verification experiments confirmed the performance efficiency of the prototype system. The experimental results showed that the proposed imaging system can capture images with high spatial and spectral resolutions under low illumination conditions. The proposed algorithm improved the spectral resolution of the acquired data from 9 to 31 bands, and the average peak signal-to-noise ratio remained above 43 dB, which is 13 dB higher than those of the state-of-the-art coded aperture snapshot spectral imaging methods. Simultaneously, the frame rate of the imaging system was up to 5000 frames/s under natural daylight.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Coded aperture snapshot spectral imaging (CASSI) has been demonstrated as a feasible solution to recover a 3D hyperspectral image by using a single 2D measurement. In this paper, we propose a new hybrid camera design for CASSI to capture high quality hyperspectral images while maintaining the snapshot advantage. Specifically, we employ a complementary RGB camera in conjunction with the CASSI system. The recorded RGB image can provide reliable spectral clue of the scene. By combining the coded hyperspectral information from the CASSI branch and the uncoded color information from the RGB branch, hyperspectral images can be reconstructed with high fidelity. Furthermore, by conducting demosaicing on the raw RGB image as a preprocessing procedure, even better performance can be achieved. Both theoretical analysis and simulation results show improved accuracy of the proposed method compared to the state-of-the-arts.

Coded aperture snapshot spectral imaging (CASSI) faces the ill‐posed inverse problem, and reconstruction accuracy remains unsatisfactory despite various regularizers. Dual‐camera CASSI (DC‐CASSI) mitigates this problem by capturing panchromatic (PAN) images alongside compressed measurements. Although untrained deep neural networks (DNN) can leverage PAN imagery, they often struggle to preserve spectral integrity and ensure optimization stability. To address these dual issues, this paper proposes a panchromatic‐guided self‐supervised framework (PGSF). Its core is a novel spectral low‐rank guided spatial enhancement module (SLGSEM), which performs dual spatial‐semantic enhancement of network inputs by fusing PAN guidance with a learned low‐rank spectral prior. This module seamlessly integrates with various untrained DNN (joint network). Furthermore, the joint network is embedded as a regularizer within an ADMM‐based optimization algorithm, enabling self‐supervised parameter learning and progressive input refinement throughout iterations. Extensive experiments demonstrate that PGSF significantly enhances untrained DNN performance, achieving state‐of‐the‐art reconstruction in both quantitative metrics and visual quality without requiring any training data.

Coded aperture snapshot spectral imaging (CASSI) is an effective tool to capture real-world 3D hyperspectral images. While a number of existing work has been conducted for hardware and algorithm design, we make a step towards the low-cost solution that enjoys video-rate high-quality reconstruction. To make solid progress on this challenging yet under-investigated task, we reproduce a stable single disperser (SD) CASSI system to gather large-scale real-world CASSI data and propose a novel deep convolutional network to carry out the real-time reconstruction by using self-attention. In order to jointly capture the self-attention across different dimensions in hyperspectral images (i.e., channel-wise spectral correlation and non-local spatial regions), we propose Spatial-Spectral Self-Attention (TSA) to process each dimension sequentially, yet in an order-independent manner. We employ TSA in an encoder-decoder network, dubbed TSA-Net, to reconstruct the desired 3D cube. Furthermore, we investigate how noise affects the results and propose to add shot noise in model training, which improves the real data results significantly. We hope our large-scale CASSI data serve as a benchmark in future research and our TSA model as a baseline in deep learning based reconstruction algorithms. Our code and data are available at https://github.com/mengziyi64/TSA-Net .

Unlike straightforward registration problems encountered in broadband imaging, spectral imaging in fielded instruments often suffers from a combination of imaging aberrations that make spatial co-registration of the images a challenging problem. Depending on the sensor architecture, typical problems to be mitigated include differing focus, magnification, and warping between the images in the various spectral bands due to optics differences; scene shift between spectral images due to parallax; and scene shift due to temporal misregistration between the spectral images. However, typical spectral images sometimes contain scene commonalities that can be exploited in traditional ways. As a first step toward automatic spatial co-registration for spectral images, we exploit manually-selected scene commonalities to produce transformation parameters in a four-channel spectral imager. The four bands consist of two mid-wave infrared channels and two short-wave infrared channels. Each of the four bands is blurred differently due to differing focal lengths of the imaging optics, magnified differently, warped differently, and translated differently. Centroid location techniques are used on the scene commonalities in order to generate sub-pixel values for the fiducial markers used in the transformation polygons, and conclusions are drawn about the effectiveness of such techniques in spectral imaging applications.