Compressive Coded Aperture Spectral Imaging: An Introduction

Coded aperture snapshot spectral imager (CASSI) has been a popular spectral imaging architecture for its ability of capturing hyperspectral images with high temporal resolution. However, such snapshot imaging system entails a large sacrifice in the spatial resolution of the data cube, since only a small amount of light gets into the imager during one snapshot. Also, the spatial resolution of the CASSI system is limited by the pixel size (and amount) of the detector, while it is difficult to fabricate a dense detector with small pixel size, especially for infrared spectral bands. Super-resolution is an advanced post-processing technique to alleviate such problem by exploiting the prior information of the image. In this letter, we try to realize image super-resolution from the perspective of developing new form of measurements by taking advantage of a modified CASSI system equipped with a coded aperture with higher spatial resolution than the detector, merging the SR model into the hardware configuration. Then the original data cube can be reconstructed from lower resolution measurements, thus the super-resolution is realized during the compressive sensing reconstruction process. The new system can be achieved based on the classical CASSI architecture in two dual ways, one by replacing the coded aperture with a higher resolution one and the other by substituting the focal plane array (FPA) detector with a lower resolution one. The experiments show that, we can recover images of higher quality with the first modification of CASSI system above, simply using a higher resolution coded aperture.

Coded aperture snapshot spectral imager (CASSI), efficiently captures 3D spectral images by sensing 2D projections of the scene. While CASSI offers a substantial reduction in acquisition time, compared to traditional scanning optical systems, it requires a reconstruction post-processing step. Furthermore, to obtain high-quality reconstructions, an accurate propagation model is required. Notably, CASSI exhibits a variant spatio-spectral sensor response, making it difficult to acquire an accurate propagation model. To address these inherent limitations, this work proposes to learn a deep non-linear fully differentiable propagation model that can be used as a regularizer within an optimization-based reconstruction algorithm. The proposed approach trains the non-linear spatially-variant propagation model using paired compressed measurements and spectral images, by employing side information only in the calibration step. From the deep propagation model incorporation into a plug-and-play alternating direction method of multipliers framework, our proposed method outperforms traditional CASSI linear-based models. Extensive simulations and a testbed implementation validate the efficacy of the proposed methodology.

Spectral imaging is of interest in many applications, including wide-area airborne surveillance, remote sensing, and tissue spectroscopy. Coded aperture spectral snapshot imaging (CASSI) provides an efficient mechanism to capture a 3D spectral cube with a single shot 2D measurement. CASSI uses a focal plane array (FPA) measurement of a spectrally dispersed, aperture coded, source. The spectral cube is then attained using a compressive sensing reconstruction algorithm. In this paper, we explore a new approach referred to as random convolution snapshot spectral imaging (RCSSI). It is based on FPA measurements of spectrally dispersed coherent sources that have been randomly convoluted by a spatial light modulator. The new method, based on the theory of compressive sensing via random convolutions, is shown to outperform traditional CASSI systems in terms of PSNR spectral image cube reconstructions.

Coded aperture snapshot spectral imager (CASSI) based on compressed sensing can capture a three-dimensional (3-D) data cube information in one snapshot without scanning, so it is a kind of novel spectral imaging technique with good prospects. In this paper, the data flow procession of two kind of CASSI were studied, which similarities and differences were depicted. The basic principle of the encoded representation of both spectral information and 2-D spatial information of a scene were presented. The numerical estimation techniques based on the compressive sensing theory were applied to reconstruct 3-D data cube.

A novel compressive 3D imaging spectrometer based on the coded aperture snapshot spectral imager (CASSI) is proposed. By inserting a microlens array (MLA) into the CASSI system, one can capture spectral data of 3D objects in a single snapshot without requiring 3D scanning. The 3D spatio-spectral sensing phenomena is modelled by computational integral imaging in tandem with compressive coded aperture spectral imaging. A set of focal stack images is reconstructed from a single compressive measurement, and presented as images focused on different depth planes where the objects are located. The proposed optical system is demonstrated with simulations and experimental results.

A MLP architecture fusing RGB and CASSI for computational spectral imaging

Conventional multispectral imaging systems based on bandpass filters struggle to record multispectral videos with high spatial resolutions because of their limited light efficiencies. This paper proposes a multi-aperture multispectral imaging system based on notch filters that overcomes this limitation by allowing light from most of the spectrum to pass through. Based on this imaging principle, a prototype multi-aperture multispectral imaging system comprising notch filters was built and demonstrated. Further, a dictionary learning- and total variation-based spectral super-resolution algorithm was developed to reconstruct spectral images. The simulation results obtained using public multispectral datasets showed that, compared to the dictionary learning-based spectral super-resolution algorithm, the proposed algorithm reconstructed the spectral information with a higher accuracy and removed noise, and the verification experiments confirmed the performance efficiency of the prototype system. The experimental results showed that the proposed imaging system can capture images with high spatial and spectral resolutions under low illumination conditions. The proposed algorithm improved the spectral resolution of the acquired data from 9 to 31 bands, and the average peak signal-to-noise ratio remained above 43 dB, which is 13 dB higher than those of the state-of-the-art coded aperture snapshot spectral imaging methods. Simultaneously, the frame rate of the imaging system was up to 5000 frames/s under natural daylight.

Coded Aperture Snapshot Spectral Imaging (CASSI) is an effective tool to capture spectral images, which has the advantages of snapshot imaging, high luminous flux, high signal-to-noise ratio and low sampling frequency. However, conventional CASSI generally uses refractive prisms or gratings for spectral dispersion, which leads to the nonlinear dispersion phenomenon and the requirement of large detector chip respectively. To overcome these issues, conventional refractive prisms or gratings are replaced by an axially dispersive diffractive optical element (DOE, i.e., computational optics) together with a RGB Bayer filter (i.e., a color-coded aperture) in this study. Specifically, the spatial-spectral information of a test scene is jointly modulated by the DOE and the Bayer filter integrated with a sensor chip. A fully differentiable imaging model is built based on the principle of diffractive optics and the deep learning technology. Furthermore, an optimization design of the DOE with the coded aperture is realized through an end-to-end approach, the output spectral images of which are restored by a Res-Unet neural network. Several simulation results show that up to 31 high-fidelity spectral bands in the range of 400 to 700 nm with a good spatial and spectral resolution can be recovered by the proposed snapshot system.

Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.

The coded aperture snapshot spectral imaging (CASSI) architecture has been employed widely for capturing hyperspectral video. Despite allowing concurrent capture of hyperspectral video, spatial modulation in CASSI sacrifices image resolution significantly while reconstructing spectral projection via sparse sampling. Several multiview alternatives have been proposed to handle this low spatial resolution problem and improve measurement accuracy, for instance, by adding a translation stage for the coded aperture or changing the static coded aperture with a digital micromirror device for dynamic modulation. State‐of‐the‐art solutions enhance spatial resolution significantly but are incapable of capturing video using CASSI. In this paper, we present a novel compressive coded aperture imaging design that increases spatial resolution while capturing 4D hyperspectral video of dynamic scenes. We revise the traditional CASSI design to allow for multiple sampling of the randomness of spatial modulation in a single frame. We demonstrate that our compressive video spectroscopy approach yields enhanced spatial resolution and consistent measurements, compared with the traditional CASSI design.

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

Coded aperture snapshot spectral imaging (CASSI) has been demonstrated as a feasible solution to recover a 3D hyperspectral image by using a single 2D measurement. In this paper, we propose a new hybrid camera design for CASSI to capture high quality hyperspectral images while maintaining the snapshot advantage. Specifically, we employ a complementary RGB camera in conjunction with the CASSI system. The recorded RGB image can provide reliable spectral clue of the scene. By combining the coded hyperspectral information from the CASSI branch and the uncoded color information from the RGB branch, hyperspectral images can be reconstructed with high fidelity. Furthermore, by conducting demosaicing on the raw RGB image as a preprocessing procedure, even better performance can be achieved. Both theoretical analysis and simulation results show improved accuracy of the proposed method compared to the state-of-the-arts.

Coded aperture snapshot spectral imaging (CASSI) faces the ill‐posed inverse problem, and reconstruction accuracy remains unsatisfactory despite various regularizers. Dual‐camera CASSI (DC‐CASSI) mitigates this problem by capturing panchromatic (PAN) images alongside compressed measurements. Although untrained deep neural networks (DNN) can leverage PAN imagery, they often struggle to preserve spectral integrity and ensure optimization stability. To address these dual issues, this paper proposes a panchromatic‐guided self‐supervised framework (PGSF). Its core is a novel spectral low‐rank guided spatial enhancement module (SLGSEM), which performs dual spatial‐semantic enhancement of network inputs by fusing PAN guidance with a learned low‐rank spectral prior. This module seamlessly integrates with various untrained DNN (joint network). Furthermore, the joint network is embedded as a regularizer within an ADMM‐based optimization algorithm, enabling self‐supervised parameter learning and progressive input refinement throughout iterations. Extensive experiments demonstrate that PGSF significantly enhances untrained DNN performance, achieving state‐of‐the‐art reconstruction in both quantitative metrics and visual quality without requiring any training data.

Coded aperture snapshot spectral imaging (CASSI) is an effective tool to capture real-world 3D hyperspectral images. While a number of existing work has been conducted for hardware and algorithm design, we make a step towards the low-cost solution that enjoys video-rate high-quality reconstruction. To make solid progress on this challenging yet under-investigated task, we reproduce a stable single disperser (SD) CASSI system to gather large-scale real-world CASSI data and propose a novel deep convolutional network to carry out the real-time reconstruction by using self-attention. In order to jointly capture the self-attention across different dimensions in hyperspectral images (i.e., channel-wise spectral correlation and non-local spatial regions), we propose Spatial-Spectral Self-Attention (TSA) to process each dimension sequentially, yet in an order-independent manner. We employ TSA in an encoder-decoder network, dubbed TSA-Net, to reconstruct the desired 3D cube. Furthermore, we investigate how noise affects the results and propose to add shot noise in model training, which improves the real data results significantly. We hope our large-scale CASSI data serve as a benchmark in future research and our TSA model as a baseline in deep learning based reconstruction algorithms. Our code and data are available at https://github.com/mengziyi64/TSA-Net .

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging. With this review paper, researchers will (1)understand the technological trends of upcoming spectral cameras, (2)understand new specific applications that portable spectral imaging unlocked, and (3)evaluate proper spectral imaging systems for their specific applications. We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition. There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings. Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.