Dictionary Learning- and Total Variation-Based High-Light-Efficiency Snapshot Multi-Aperture Spectral Imaging

A MLP architecture fusing RGB and CASSI for computational spectral imaging

Coded aperture snapshot spectral imager (CASSI) has been a popular spectral imaging architecture for its ability of capturing hyperspectral images with high temporal resolution. However, such snapshot imaging system entails a large sacrifice in the spatial resolution of the data cube, since only a small amount of light gets into the imager during one snapshot. Also, the spatial resolution of the CASSI system is limited by the pixel size (and amount) of the detector, while it is difficult to fabricate a dense detector with small pixel size, especially for infrared spectral bands. Super-resolution is an advanced post-processing technique to alleviate such problem by exploiting the prior information of the image. In this letter, we try to realize image super-resolution from the perspective of developing new form of measurements by taking advantage of a modified CASSI system equipped with a coded aperture with higher spatial resolution than the detector, merging the SR model into the hardware configuration. Then the original data cube can be reconstructed from lower resolution measurements, thus the super-resolution is realized during the compressive sensing reconstruction process. The new system can be achieved based on the classical CASSI architecture in two dual ways, one by replacing the coded aperture with a higher resolution one and the other by substituting the focal plane array (FPA) detector with a lower resolution one. The experiments show that, we can recover images of higher quality with the first modification of CASSI system above, simply using a higher resolution coded aperture.

Coded aperture snapshot spectral imager (CASSI), efficiently captures 3D spectral images by sensing 2D projections of the scene. While CASSI offers a substantial reduction in acquisition time, compared to traditional scanning optical systems, it requires a reconstruction post-processing step. Furthermore, to obtain high-quality reconstructions, an accurate propagation model is required. Notably, CASSI exhibits a variant spatio-spectral sensor response, making it difficult to acquire an accurate propagation model. To address these inherent limitations, this work proposes to learn a deep non-linear fully differentiable propagation model that can be used as a regularizer within an optimization-based reconstruction algorithm. The proposed approach trains the non-linear spatially-variant propagation model using paired compressed measurements and spectral images, by employing side information only in the calibration step. From the deep propagation model incorporation into a plug-and-play alternating direction method of multipliers framework, our proposed method outperforms traditional CASSI linear-based models. Extensive simulations and a testbed implementation validate the efficacy of the proposed methodology.

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

Coded Aperture Snapshot Spectral Imaging (CASSI) is an effective tool to capture spectral images, which has the advantages of snapshot imaging, high luminous flux, high signal-to-noise ratio and low sampling frequency. However, conventional CASSI generally uses refractive prisms or gratings for spectral dispersion, which leads to the nonlinear dispersion phenomenon and the requirement of large detector chip respectively. To overcome these issues, conventional refractive prisms or gratings are replaced by an axially dispersive diffractive optical element (DOE, i.e., computational optics) together with a RGB Bayer filter (i.e., a color-coded aperture) in this study. Specifically, the spatial-spectral information of a test scene is jointly modulated by the DOE and the Bayer filter integrated with a sensor chip. A fully differentiable imaging model is built based on the principle of diffractive optics and the deep learning technology. Furthermore, an optimization design of the DOE with the coded aperture is realized through an end-to-end approach, the output spectral images of which are restored by a Res-Unet neural network. Several simulation results show that up to 31 high-fidelity spectral bands in the range of 400 to 700 nm with a good spatial and spectral resolution can be recovered by the proposed snapshot system.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

It has been noticed that spectral information can be used for analyzing and separating different biological tissues. However, most of the studies for spectral image acquisitions are mainly done in vitro. Usually the main restrictions for in vivo measurements are the size or the weight of the spectral camera. If the camera weights too much, the surgery microscope cannot be stabilized. If the size of the camera is too big, it will disturb the surgeon or even risk the safety of the patient. The main goal of this study was to develop an independent spectral imaging device which can be used for collecting spectral information from the neurosurgeries without any previously described restrictions. Size of the imaging system is small enough not to disturb the surgeon during the surgery. The developed spectral imaging system is used for collecting a spectral database which can be used for the future imaging systems.

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

Spectral imaging has been proved as a promising technology to perform scientific documentation and analysis of cultural heritage objects. Imaging systems used for this purpose varies in different forms and complexity. Most of the spectral devices are designed not specifically for cultural heritage imaging, but for other application domains like remote sensing, satellite imaging etc. While these imaging systems used for cultural heritage scanning, the same processing workflow may not be adequate to meet the required image quality. In this paper, we investigate one of the several quality parameters, geometrical distortions in a spectral image caused by the translation stage of the camera. For this study, we used a hyperspectral image dataset derived from a pushbroom hyperspectral imaging system attached in a rotational translator. Using geometrical model, we have estimated the distortion, which is function of the scanning angle and distance. Image correction has been proposed and tested over a number of images acquired at the laboratory and in a museum environment.

This work is focused on the study and comparison of the performance for color measurements of different systems based on optoelectronic imaging sensors. We used two different configurations of the imaging system, one with three acquisition channels and the other with more spectral bands, in order to measure the color associated to each pixel of the captured scene. We applied different methodologies to obtain the XYZ tristumulus values from the measured digital signals. The different techniques included an absolute spectral and colorimetric characterization of the system and also direct transformations between both sets, which used several mathematical fittings such as the pseudo-inverse technique, a non-linear estimation method and the principal component analysis. The proposed configurations were experimentally tested imaging the patches of the Gretagmacbeth ColorChecker DC and Color Rendition charts placed in a light booth, and measuring the corresponding colors. The results obtained showed that optoelectronic imaging systems can be used in order to perform rather accurate color measurements with high spatial resolution. Specifically, the best results in terms of CIELab color differences were achieved by using a multispectral configuration of the imaging system with seven spectral bands and directly transforming the digital signals into XYZ tristimulus values by means of the pseudo-inverse technique.

Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

Coded aperture snapshot spectral imager (CASSI) based on compressed sensing can capture a three-dimensional (3-D) data cube information in one snapshot without scanning, so it is a kind of novel spectral imaging technique with good prospects. In this paper, the data flow procession of two kind of CASSI were studied, which similarities and differences were depicted. The basic principle of the encoded representation of both spectral information and 2-D spatial information of a scene were presented. The numerical estimation techniques based on the compressive sensing theory were applied to reconstruct 3-D data cube.

A novel compressive 3D imaging spectrometer based on the coded aperture snapshot spectral imager (CASSI) is proposed. By inserting a microlens array (MLA) into the CASSI system, one can capture spectral data of 3D objects in a single snapshot without requiring 3D scanning. The 3D spatio-spectral sensing phenomena is modelled by computational integral imaging in tandem with compressive coded aperture spectral imaging. A set of focal stack images is reconstructed from a single compressive measurement, and presented as images focused on different depth planes where the objects are located. The proposed optical system is demonstrated with simulations and experimental results.

Coded aperture snapshot spectral imaging (CASSI) has been demonstrated as a feasible solution to recover a 3D hyperspectral image by using a single 2D measurement. In this paper, we propose a new hybrid camera design for CASSI to capture high quality hyperspectral images while maintaining the snapshot advantage. Specifically, we employ a complementary RGB camera in conjunction with the CASSI system. The recorded RGB image can provide reliable spectral clue of the scene. By combining the coded hyperspectral information from the CASSI branch and the uncoded color information from the RGB branch, hyperspectral images can be reconstructed with high fidelity. Furthermore, by conducting demosaicing on the raw RGB image as a preprocessing procedure, even better performance can be achieved. Both theoretical analysis and simulation results show improved accuracy of the proposed method compared to the state-of-the-arts.