Rank Minimization Code Aperture Design for Spectrally Selective Compressive Imaging

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

Snapshot Compressive Spectral Imaging Systems (SCSI) compress the scenes by capturing 2D projections of the encoded underlying signals. A decoder, trained with pre-acquired datasets, reconstructs the spectral images. SCSI systems based on diffractive optical elements (DOE) provide a small form factor and the single DOE can be optimized in an end-to-end manner. Since the spectral image is highly compressed in a SCSI system based on a single DOE, the quality of image reconstruction can be insufficient for diverse spectral imaging applications. This work proposes a multishot spectral imaging system employing a double-phase encoding with a double DOE architecture (DoDo), to improve the spectral reconstruction performance. The first DOE is fixed and provides the benefits of the diffractive optical systems. The second DOE provides the variable encoding of the multishot architectures. The work presents a differentiable mathematical model for the multishot DoDo system and optimizes the parameters of the DoDo architecture in an end-to-end manner. The proposed system was tested using simulations and a hardware prototype. To obtain a low-cost system, the implementation uses a deformable mirror for the variable DOE. The proposed DoDo system shows an improvement of up to 4dB in PSNR in the reconstructed spectral images compared with the single DOE system.

A MLP architecture fusing RGB and CASSI for computational spectral imaging

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

LADMM-Net: An unrolled deep network for spectral image fusion from compressive data

Hyperspectral and multispectral imaging for evaluating food safety and quality

Compressive spectral imaging (CSI) acquires compressed observations of a spectral scene by applying different coding patterns at each spatial location and then performing a spectral-wise integration. Relying on compressive sensing, spectral image reconstruction is achieved by using nonlinear and relatively expensive optimization-based algorithms. In the CSI literature, several works have focused on improving reconstructions quality by properly designing the set of coding patterns. However, signal recovery is not actually necessary in many signal processing applications. For instance, assuming that compressed measurements with similar characteristics lie on the same subspace, unsupervised methods such as subspace clustering can be used to separate them into the same cluster. Since the structure of compressed measurements is defined by the applied codification, it is possible to improve clustering performance. This paper proposes to design a set of coding patterns such that inter-class and intra-class data structure is preserved after the CSI acquisition in order to improve clustering results directly on the compressed domain. To validate the coding pattern design, an algorithm based on sparse subspace clustering (SSC) is proposed to perform clustering on the compressed measurements. The proposed algorithm adds a three-dimensional (3-D) spatial regularizer to the SSC problem exploiting the spatial correlation of spectral images. In general, an overall accuracy up to 83.81% is obtained, when noisy measurements are assumed. In addition, a difference of at most 4% in terms of overall accuracy was observed when comparing the clustering results obtained by the full 3-D data with those achieved using CSI measurements acquired with the proposed coding pattern design.

Spectrum, as the 'fingerprint' of materials, reveals their composition and properties. However, traditional spectral imaging systems are hindered by their bulkiness, high cost, time-consuming and high complexity, limiting their widespread use. By combining quantum dot (QD) spectral sensing with multi-spectral filter array designs, QD mosaic snapshot spectral imaging provides a real-time, compact, low-cost, and low-complexity alternative to traditional spectral imaging systems. Yet, the high flexibility of QD response curves presents a significant challenge: how to achieve an efficient, principle-guided, and data-adaptive design of high reconstruction performance? In this work, we propose a two-stage design method. Firstly, a set of absorption spectra with distinct particle sizes is selected using QR decomposition. Then, by leveraging the similarity between the imaging model and the convolutional operation, we optimize the concentration of QDs via gradient descent during the training stage of the reconstruction network, which serves as the software decoder to recover the spectral images from the encoded measurements. The proposed method is validated on the CAVE and Harvard datasets across multiple cases, achieving up to a 6.82% improvement in PSNR over the baseline, along with consistent gains in SSIM and SAM metrics. These results confirm the effectiveness of the proposed principle-guided method in achieving high reconstruction performance for spectral imaging systems.

In compressive sensing (CS), the restricted isometry property (RIP) is an important condition on measurement matrices which guarantees the recovery of sparse signals with undersampled measurements. It has been proved in the prior works that both random (e.g., i.i.d. Gaussian, Bernoulli, $\ldots$ ) and Toeplitz matrices satisfy the RIP with high probability. However, structured matrices, such as banded Toeplitz matrices have drawn more attention since their structures have the advantage of fast matrix multiplication which may decrease the computational complexity of recovery algorithms. In this paper, we show that banded block Toeplitz matrices satisfy the RIP condition with high probability. Banded block Toeplitz matrices can be used in the sparse multi-channel source separation. The banded block Toeplitz matrices decrease the computational complexity while they have fewer number of non-zero entries in comparison to the same dimensional banded Toeplitz matrices. Furthermore, our simulation results show that banded block Toeplitz matrices outperform banded Toeplitz matrices in signal estimation. The analytical RIP bound for banded block Toeplitz matrices is provided in this paper and the RIP bound of sparse Gaussian matrices is also obtained as an upper bound for banded block Toeplitz matrices. Our simulation and analytical results show that sparse Gaussian random matrices do satisfy the RIP condition with high probability. The probability of satisfying the RIP depends on the probability of zero entries.

We combined a homebuilt multiphoton microscope and a prism-CCD based spectrograph to develop a spectral imaging system capable of imaging deep into live tissues. The spectral images originate from the two-photon autofluorescence of the tissue and second harmonic signal from the collagen fibers. A highly penetrating near-infrared light is used to excite the endogenous fluorophores via multiphoton excitation enabling us to produce high quality images deep into the tissue. We were able to produce 100-channel (330 nm to 600 nm) autofluorescence spectral images of live skin tissues in less than 2 minutes for each xy-section. The spectral images rendered in RGB (real) colors showed green hair shafts, blue cells, and purple collagen. Analysis on the optical signal degradation with increasing depth of the collagen second-harmonic signal showed 1) exponential decay behavior of the intensity and 2) linear broadening of the spectrum. This spectral imaging system is a promising tool for both in biological applications and biomedical applications such as optical biopsy.

The capability of simultaneously capturing spatial and spectral information represents the most distinctive advantage of spectral imaging technology. To expand its applicability to different scenarios, compact and cost-effective spectral imaging systems are necessary. The mosaic spectral imaging sensor is considered a promising technical pathway for future consumer-grade video spectral imaging, as it offers snapshot acquisition along with advantages such as compactness and integration flexibility. However, this technology heavily relies on spectral reconstruction algorithms, many of which suffer from either excessive computational latency or insufficient robustness to noise, so that their practical utility is limited. In this study, we developed a noise calibration protocol for mosaic spectral sensors based on the sensor noise theory as well as a high-fidelity DN simulation algorithm. These efforts enable the efficient construction of a massive and realistic DN-spectrum dataset. Based on this dataset, we further proposed a spectral reconstruction model using a two-dimensional convolutional neural network (2D-CNN). As a result, the newly developed model demonstrates a strong noise robustness and balanced capabilities in reconstructing the samples with substantial high-level noise, and thus achieves an average reconstruction fidelity of 98.37%, which maintains a level of performance that is comparable to or exceeds that of high-fidelity and complex traditional algorithms. The reconstruction RMSE ratio is only 1.87%, which is only half of that of traditional algorithms. Moreover, the full-image reconstruction time is less than 1 second, which is significantly faster than current high-fidelity algorithms that often require tens of minutes. We believe these advancements will enable lightweight spectral imaging systems to achieve accurate real-time spectral imaging in wide scenarios or under harsher lighting conditions.

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

Long-wave infrared (LWIR) spectral imaging plays a critical role in various applications such as gas monitoring, mineral exploration, and fire detection. Recent advancements in computational spectral imaging, powered by advanced algorithms, have enabled the acquisition of high-quality spectral images in real time, such as with the Uncooled Snapshot Infrared Spectrometer (USIRS). However, the USIRS system faces challenges, particularly a low spectral resolution and large amount of data noise, which can degrade the image quality. Deep learning has emerged as a promising solution to these challenges, as it is particularly effective at handling noisy data and has demonstrated significant success in hyperspectral imaging tasks. Nevertheless, the application of deep learning in LWIR imaging is hindered by the severe scarcity of long-wave hyperspectral image data, which limits the training of robust models. Moreover, existing networks that rely on convolutional layers or attention mechanisms struggle to effectively capture both local and global spectral correlations. To address these limitations, we propose the pixel-based Hierarchical Spectral Transformer (HST), a novel deep learning architecture that learns from publicly available single-pixel long-wave infrared spectral databases. The HST is designed to achieve a high spectral resolution for LWIR spectral image reconstruction, enhancing both the local and global contextual understanding of the spectral data. We evaluated the performance of the proposed method on both simulated and real-world LWIR data, demonstrating the robustness and effectiveness of the HST in improving the spectral resolution and mitigating noise, even with limited data.

If multiple objects are located closely, we often observe interreflection on the object surfaces. The interreflection is accompanied by a change in the appearance of the object surfaces. In this paper, we propose a method to estimate the spectral image components from the captured images of two closely apposed objects. A spectral imaging system is used for image capturing of both objects under uniform illumination. First, we describe the basis bispectral characteristics of a fluorescent object. Second, we model the interreflection of two objects with the bispectral characteristics. The surface appearance of each object was based on both reflected component and luminescent component. We show that the spectral composition of interreflection is determined by a series of multiplication of the Donaldson matrices of the two objects. The spectral components of one bounce interreflection can be limited to only four spectral functions. Third, an algorithm is developed to estimate the spectral image components from the observed spectral radiance images that are influenced by the interreflection. The algorithm is based on a two-step estimation procedure in which the spectral components are estimated separately in the outside and inside of the emission wavelength range. The feasibility of the proposed method is investigated using fluorescent samples in details.

DMP-Net: Deep semantic prior compressed spectral reconstruction method towards intraoperative imaging of brain tissue.