Spectral-Spatial Classification from Multi-Sensor Compressive Measurements Using Superpixels

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

Compressive spectral imaging (CSI) acquires compressed observations of a spectral scene by applying different coding patterns at each spatial location and then performing a spectral-wise integration. Relying on compressive sensing, spectral image reconstruction is achieved by using nonlinear and relatively expensive optimization-based algorithms. In the CSI literature, several works have focused on improving reconstructions quality by properly designing the set of coding patterns. However, signal recovery is not actually necessary in many signal processing applications. For instance, assuming that compressed measurements with similar characteristics lie on the same subspace, unsupervised methods such as subspace clustering can be used to separate them into the same cluster. Since the structure of compressed measurements is defined by the applied codification, it is possible to improve clustering performance. This paper proposes to design a set of coding patterns such that inter-class and intra-class data structure is preserved after the CSI acquisition in order to improve clustering results directly on the compressed domain. To validate the coding pattern design, an algorithm based on sparse subspace clustering (SSC) is proposed to perform clustering on the compressed measurements. The proposed algorithm adds a three-dimensional (3-D) spatial regularizer to the SSC problem exploiting the spatial correlation of spectral images. In general, an overall accuracy up to 83.81% is obtained, when noisy measurements are assumed. In addition, a difference of at most 4% in terms of overall accuracy was observed when comparing the clustering results obtained by the full 3-D data with those achieved using CSI measurements acquired with the proposed coding pattern design.

Massive amounts of data in spectral imagery increase acquisition, storing and processing costs. Compressive spectral imaging (CSI) methods allow the reconstruction of spatial and spectral information from a small set of random projections. The single pixel camera is a low cost optical architecture which enables the compressive acquisition of spectral images. Traditional CSI reconstruction methods obtain a sparse approximation of the underlying spatial and spectral information, however the complexity of these algorithms increases in proportion to the dimensionality of the data. This work proposes a multiresolution (MR) CSI reconstruction approach from single pixel camera measurements that exploits spectral similarities between pixels to group them in super-pixels such that the total number of unknowns in the inverse problem is reduced. Specifically, two different types of super-pixels are considered: rectangular and irregular structures. Simulation and experimental results show that the proposed MR scheme improves reconstruction quality in up to 6dB of PSNR and reconstruction time in up to 90% with respect to the traditional full resolution reconstructions.

LADMM-Net: An unrolled deep network for spectral image fusion from compressive data

Compressive spectral imaging (CSI) allows the acquisition of the spectral information of a three-dimensional scenes by using coded projections in a sensor with lower dimension. However, the compressed sampling of information with simultaneously high spatial and high spectral resolution demands expensive high-resolution sensors. One of the main challenges in CSI is to obtain a high-quality image of high-resolution reconstructions using low-cost architectures. Single pixel camera is an approach that has had a high impact in spectroscopy, due to its low-cost implementation compared to CSI architectures with 2D sensors. On the other hand, recent works have been shown that image fusion using measurements from a CSI sensor based on side information leads to improvement in the quality of the fused image. This work proposes a spectral image fusion methodology for increasing the spatio-spectral resolution through side information and at the same time improve the reconstruction quality of the data cube with a low-cost architecture, optimizing the similarity of the reconstructed spectral image with each sensor. Simulations and experimental results for the proposed methodology show that improve the quality of the reconstruction in up to 11 dB with respect to the traditional approach of upsampling the single pixel image reconstruction through bilinear interpolation.

Compressive spectral imaging has garnered significant attention for its ability to effectively enhance the captured spatial and spectral information. Predominant methods, based on compressive sensing, typically formulate the imaging task as a constrained optimization problem and rely on hand-crafted priors to model the sparsity of spectral images. However, these approaches often suffer from suboptimal performance due to the inherent difficulty of identifying an appropriate transform space where spectral images exhibit sparsity. To overcome this limitation, we propose a novel convolutional sparse coding-inspired untrained network prior for fast and adaptive identification of the sparse transform domain and compressible signal. Specifically, a Lightweight Convolutional Thresholding sparse Coding (LCTC) network is designed as the sparse transform domain, with its inputs interpreted as sparse coefficients. Crucially, both the transform domain and its coefficients are solved in a self-supervised learning manner. Furthermore, we demonstrate that LCTC prior can be seamlessly incorporated into the iterative optimization algorithm as a Plug-and-Play (PnP) regularization. Both the LCTC and PnP-LCTC exhibit superior performance compared to previous methods. Experiments under various scenarios validate the effectiveness and efficiency of our approach.

DMP-Net: Deep semantic prior compressed spectral reconstruction method towards intraoperative imaging of brain tissue.

Compressive spectral imaging (CSI) enables the acquisition of spectral and spatial information of a scene using fewer projected measurements than traditional scanning approaches. Recently, research efforts have focused on obtaining high-resolution spectral images via expensive detectors and sophisticated CSI devices. Alternatively, high-resolution spectral images can be obtained using side information or fusion of compressed measurements, without significantly increasing acquisition costs. Indeed, these approaches retrieve improved resolution images applying iterative and computationally expensive algorithms. This paper proposes the fusion of compressed measurements obtained from two state-of-the-art CSI systems, the single-pixel camera (SPC) and the three-dimensional coded aperture snapshot imaging system (3D-CASSI), such that high-resolution images can be obtained by exploiting detailed spectra provided by the SPC and high spatial resolution of the 3D-CASSI. Specifically, a noniterative reconstruction algorithm is proposed, based on the fact that the spatial-spectral data lie in a low-dimensional subspace. In contrast to related works, the proposed approach relies on implementable CSI systems. Simulations and experimental results show the effectiveness of the proposed method compared to similar approaches, both in reconstruction quality and complexity. Specifically, the proposed method is up to 5.6 times faster than its counterparts and provides comparable quality of attained reconstructions.

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

Snapshot Compressive Spectral Imaging Systems (SCSI) compress the scenes by capturing 2D projections of the encoded underlying signals. A decoder, trained with pre-acquired datasets, reconstructs the spectral images. SCSI systems based on diffractive optical elements (DOE) provide a small form factor and the single DOE can be optimized in an end-to-end manner. Since the spectral image is highly compressed in a SCSI system based on a single DOE, the quality of image reconstruction can be insufficient for diverse spectral imaging applications. This work proposes a multishot spectral imaging system employing a double-phase encoding with a double DOE architecture (DoDo), to improve the spectral reconstruction performance. The first DOE is fixed and provides the benefits of the diffractive optical systems. The second DOE provides the variable encoding of the multishot architectures. The work presents a differentiable mathematical model for the multishot DoDo system and optimizes the parameters of the DoDo architecture in an end-to-end manner. The proposed system was tested using simulations and a hardware prototype. To obtain a low-cost system, the implementation uses a deformable mirror for the variable DOE. The proposed DoDo system shows an improvement of up to 4dB in PSNR in the reconstructed spectral images compared with the single DOE system.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Coded aperture snapshot spectral imager (CASSI), efficiently captures 3D spectral images by sensing 2D projections of the scene. While CASSI offers a substantial reduction in acquisition time, compared to traditional scanning optical systems, it requires a reconstruction post-processing step. Furthermore, to obtain high-quality reconstructions, an accurate propagation model is required. Notably, CASSI exhibits a variant spatio-spectral sensor response, making it difficult to acquire an accurate propagation model. To address these inherent limitations, this work proposes to learn a deep non-linear fully differentiable propagation model that can be used as a regularizer within an optimization-based reconstruction algorithm. The proposed approach trains the non-linear spatially-variant propagation model using paired compressed measurements and spectral images, by employing side information only in the calibration step. From the deep propagation model incorporation into a plug-and-play alternating direction method of multipliers framework, our proposed method outperforms traditional CASSI linear-based models. Extensive simulations and a testbed implementation validate the efficacy of the proposed methodology.

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

Spectral imaging has been extensively applied in many fields, including agriculture, environmental monitoring, biomedical diagnostics, etc. Thanks to the advances in sensor technology, spectral imaging systems nowadays provide finer and finer spectral resolution needed to characterize the spectral properties of materials. The high spectral resolution, however, raises an issue as the difference in spectral information between two adjacent wavelength bands is typically very small. As a result, much of the data in a scene seems to be redundant. However, critical information is embedded that often can be used to identify materials. This thesis aims at facilitating the analysis in spectral imaging by making use of pattern recognition techniques, on the one hand, to improve visualization, and on the other hand, to directly solve classification problems.

Hyperspectral imaging (HSI) shows great promise for the detection and classification of several diseases, particularly in the fields of optical biopsy as applied to oncology, and functional retinal imaging in ophthalmology. In this paper, we discuss the application of HSI to the detection of retinal diseases and technological solutions that address some of the fundamental difficulties of spectral imaging within the eye. HSI of the retina offers a route to non-invasively deduce biochemical and metabolic processes within the retina. For example it shows promise for the mapping of retinal blood perfusion using spectral signatures of oxygenated and deoxygenated hemoglobin. Compared with other techniques using just a few spectral measurements, it offers improved classification in the presence of spectral cross-contamination by pigments and other components within the retina. There are potential applications for this imaging technique in the investigation and treatment of the eye complications of diabetes, and other diseases involving disturbances to the retinal, or optic-nerve-head circulation. It is well known that high-performance HSI requires high signal-to-noise ratios (SNR) whereas the application of any imaging technique within the eye must cope with the twin limitations of the small numerical aperture provided by the entrance pupil to the eye and the limit on the radiant power at the retina. We advocate the use of spectrally-multiplexed spectral imaging techniques (the traditional filter wheel is a traditional example). These approaches enable a flexible approach to spectral imaging, with wider spectral range, higher SNRs and lower light intensity at the retina than could be achieved using a Fourier-transform (FT) approach. We report the use of spectral imaging to provide calibrated spectral albedo images of healthy and diseased retinas and the use of this data for screening purposes. These images clearly demonstrate the ability to distinguish between oxygenated and deoxygenated hemoglobin using spectral imaging and this shows promise for the early detection of various retinopathies.