Dual camera snapshot high-resolution-hyperspectral imaging system with parallel joint optimization via physics-informed learning.

A MLP architecture fusing RGB and CASSI for computational spectral imaging

Hyperspectral remote sensing often captures imagery where the spectral profiles of the spatial pixels are the result of the reflectance contribution of numerous materials. Spectral unmixing is then used to extract the collection of materials, or endmembers, contained in the measured spectra, and a set of corresponding fractions that indicate the abundance of each material present at each pixel. This work aims at developing a spectral unmixing algorithm directly from compressive measurements acquired using the coded-aperture snapshot spectral imaging (CASSI) system. The proposed method first uses the compressive measurements to find a sparse vector representation of each pixel in a 3-D dictionary formed by a 2-D wavelet basis and a known spectral library of endmembers. The sparse vector representation is estimated by solving a sparsity-constrained optimization problem using an algorithm based on the variable splitting augmented Lagrangian multipliers method. The performance of the proposed spectral unmixing method is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The optimal coded apertures are designed such that the CASSI sensing matrix satisfies a Restricted Isometry Property (RIP) with high probability. Simulations with synthetic hyperspectral cubes illustrate the accuracy of the proposed unmixing method.

In the coded aperture snapshot spectral imaging (CASSI) system, the coded and compressed single-channel measurements need to be reconstructed into hyperspectral cubes. Existing discriminative models reconstruct the spectral cube by optimizing the mean squared error (MSE) between the ground truth and the predicted image, employing peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as metrics to gauge the quality of reconstruction. However, these indicators often possess significant limitations in mimicking human visual perception and in discerning the impact of image distortions on perceived visual quality. In this article, a new model named CASSIDiff is proposed to reconstruct CASSI measurements, achieving advanced results in perceptual loss-based evaluation metrics such as learned perceptual image patch similarity (LPIPS) and Fréchet inception distance (FID). The diffusion model, which enjoys high accuracy and reliability in generative tasks, is used for the first time for the hyperspectral reconstruction task. A feature fusion mechanism based on discrete wavelet transform (DWT) is used to weaken the noise interference effect in the conditional diffusion model. Considering the interspectra similarity and long-range dependencies of hyperspectral data, the spatial-spectral attention mechanism is also introduced. Experiments show that CASSIDiff not only outperforms most existing algorithms in simulation datasets but also shows robustness to real data published and collected in our home-built CASSI system. The code and models are publicly available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YifanSi/CASSIDiff</uri>.

Raman spectroscopy is an efficient method for detection of explosives even in small quantities. A laser can be combined with a Coded Aperture Snapshot Spectral Imaging (CASSI) system to collect Raman spectra from a surface at stand-off distances. The CASSI-system decrease the data collection time but instead increase the reconstruction time for the Raman image. Reconstruction of Raman spectra from an ensemble of compressed sensing measurements using standard reconstruction methods such as Total Variation (TV) is rather time consuming and limits the application domain for the technique. Novel machine learning approaches such as Deep Learning (DL) has lately been applied to reconstruction problems. We evaluate our earlier developed DL approach for reconstruction of Raman spectra from an ensemble of measurements formulated as a regression problem. The DL network is trained by minimizing a loss-function which is composed of two components: a reconstruction error and a re-projection error. The evaluated method is trained on simulated data where the training data has been generated using a transfer function. The transfer function has been developed to mimic the optical properties of a CASSI system. The DL network has been trained on different training sets with different levels of background noise, different number of materials in the scene and different spatial configurations of the materials in the scene. The reconstruction results using the DL network has been qualitatively evaluated on simulated data and the results are also compared to the Two-Step Iterative Shrinkage/Thresholding (TwIST) algorithm in terms of reconstruction quality and computation time. The reconstruction time for the DL are orders of magnitude lower than for TwIST without reducing the quality of the reconstructed Raman spectra.

Coded aperture snapshot spectral imager (CASSI) has been a popular spectral imaging architecture for its ability of capturing hyperspectral images with high temporal resolution. However, such snapshot imaging system entails a large sacrifice in the spatial resolution of the data cube, since only a small amount of light gets into the imager during one snapshot. Also, the spatial resolution of the CASSI system is limited by the pixel size (and amount) of the detector, while it is difficult to fabricate a dense detector with small pixel size, especially for infrared spectral bands. Super-resolution is an advanced post-processing technique to alleviate such problem by exploiting the prior information of the image. In this letter, we try to realize image super-resolution from the perspective of developing new form of measurements by taking advantage of a modified CASSI system equipped with a coded aperture with higher spatial resolution than the detector, merging the SR model into the hardware configuration. Then the original data cube can be reconstructed from lower resolution measurements, thus the super-resolution is realized during the compressive sensing reconstruction process. The new system can be achieved based on the classical CASSI architecture in two dual ways, one by replacing the coded aperture with a higher resolution one and the other by substituting the focal plane array (FPA) detector with a lower resolution one. The experiments show that, we can recover images of higher quality with the first modification of CASSI system above, simply using a higher resolution coded aperture.

Conventional multispectral imaging systems based on bandpass filters struggle to record multispectral videos with high spatial resolutions because of their limited light efficiencies. This paper proposes a multi-aperture multispectral imaging system based on notch filters that overcomes this limitation by allowing light from most of the spectrum to pass through. Based on this imaging principle, a prototype multi-aperture multispectral imaging system comprising notch filters was built and demonstrated. Further, a dictionary learning- and total variation-based spectral super-resolution algorithm was developed to reconstruct spectral images. The simulation results obtained using public multispectral datasets showed that, compared to the dictionary learning-based spectral super-resolution algorithm, the proposed algorithm reconstructed the spectral information with a higher accuracy and removed noise, and the verification experiments confirmed the performance efficiency of the prototype system. The experimental results showed that the proposed imaging system can capture images with high spatial and spectral resolutions under low illumination conditions. The proposed algorithm improved the spectral resolution of the acquired data from 9 to 31 bands, and the average peak signal-to-noise ratio remained above 43 dB, which is 13 dB higher than those of the state-of-the-art coded aperture snapshot spectral imaging methods. Simultaneously, the frame rate of the imaging system was up to 5000 frames/s under natural daylight.

Development of energy correction based multishot snapshot spectral imaging system

This work describes numerical methods for the joint reconstruction and segmentation of spectral images taken by compressive sensing coded aperture snapshot spectral imagers (CASSI). In a snapshot, a CASSI captures a two-dimensional (2D) array of measurements that is an encoded representation of both spectral information and 2D spatial information of a scene, resulting in significant savings in acquisition time and data storage. The reconstruction process decodes the 2D measurements to render a three-dimensional spatio-spectral estimate of the scene and is therefore an indispensable component of the spectral imager. In this study, we seek a particular form of the compressed sensing solution that assumes spectrally homogeneous segments in the two spatial dimensions, and greatly reduces the number of unknowns, often turning the underdetermined reconstruction problem into one that is overdetermined. Numerical tests are reported on both simulated and real data representing compressed measurements.

Coded Aperture Snapshot Spectral Imaging (CASSI) is an effective tool to capture spectral images, which has the advantages of snapshot imaging, high luminous flux, high signal-to-noise ratio and low sampling frequency. However, conventional CASSI generally uses refractive prisms or gratings for spectral dispersion, which leads to the nonlinear dispersion phenomenon and the requirement of large detector chip respectively. To overcome these issues, conventional refractive prisms or gratings are replaced by an axially dispersive diffractive optical element (DOE, i.e., computational optics) together with a RGB Bayer filter (i.e., a color-coded aperture) in this study. Specifically, the spatial-spectral information of a test scene is jointly modulated by the DOE and the Bayer filter integrated with a sensor chip. A fully differentiable imaging model is built based on the principle of diffractive optics and the deep learning technology. Furthermore, an optimization design of the DOE with the coded aperture is realized through an end-to-end approach, the output spectral images of which are restored by a Res-Unet neural network. Several simulation results show that up to 31 high-fidelity spectral bands in the range of 400 to 700 nm with a good spatial and spectral resolution can be recovered by the proposed snapshot system.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

The application of confoca]. imaging systems to image the cornea and the ocular lens of the living eye is a major improvement in ophthalmic imaging. This new application of confocal imaging to the living eye has potential for both basic research and clinical diagnostics. The standard optical imaging systems have poor resolution and low contrast when applied to ocular tissue. The use of confocal systems to image the nearly transparent cornea and ocular lens has several advantages as an imaging system. The two important physical improvements are: (1) almost total rejection of out- offocal plane reflected light, and (2) increased lateral resolution. The problems of ocular imaging with standard slit lamps and specular microscopes include: low contrast images due to contributions to the image from out of focal plane reflections, and low resolution. Confocal imaging systems can image living eyes. The images show sharp, high resolution, high contrast features of submicron structures. Examples of ocular confocal images include the following: submicron optical sections through the various cells in the cornea, nerve fibers, cellular processes, and images through the nucleus of living cells comprising the cornea and the ocular lens. The ocular lens has been imaged. The lens capsule, lens epithelial cells, lens fibers and nuclei are readily observed. The use of realtime confocal imaging systems in ocular medical imaging has resulted in improved quality (resolution and contrast) of cellular images of living eyes. Further developments require the availability of long free working distance objectives with a high numerical aperture. This paper demonstrates the advantages of imaging ocular tissue with a confocal imaging system. Confocal imaging of ophthalmic structures may become a standard clinical tool with the new developments in instrumentation.

A new code aperture design framework for multiframe code aperture snapshot spectral imaging (CASSI) system is presented. It aims at the optimization of code aperture sets such that a group of compressive spectral measurements is constructed, each with information from a specific subset of bands. A matrix representation of CASSI is introduced that permits the optimization of spectrally selective code aperture sets. Furthermore, each code aperture set forms a matrix such that rank minimization is used to reduce the number of CASSI shots needed. Conditions for the code apertures are identified such that a restricted isometry property in the CASSI compressive measurements is satisfied with higher probability. Simulations show higher quality of spectral image reconstruction than that attained by systems using Hadamard or random code aperture sets.

A triage test to assist clinical decision-making on choosing primary chemoradiation for cervical carcinomas or primary surgery for endometrial carcinomas is important. To develop and validate a multiparametric prediction model based on MR imaging and spectroscopy in distinguishing adenocarcinomas of uterine cervical or endometrial origin. Prospective diagnostic accuracy study. Eighty-seven women: 25 cervical and 62 endometrial adenocarcinomas divided into training (n = 43; cervical/endometrial adenocarcinomas = 11/32) and validation (n = 44; 14/30) datasets. The 3T diffusion-weighted (DW) MR imaging and MR spectroscopy. Morphology, volumetric DW MR imaging and spectroscopy (MDS) scoring system with total points 0-5, based on presence of the following MR features assessed independently by two radiologists: (a) epicenter at the cervix, (b) rim enhancement, (c) disrupted cervical stromal integrity, (d) mean volumetric apparent diffusion coefficient values (ADCmean) higher than 0.98 × 10-3 mm2 /s, (e) fatty acyl δ 1.3 ppm more than 161.92 mM. Histopathology as gold standard. Logistic regression and receiver operator characteristic (ROC) curves analysis. For both the training and validation datasets, the MDS score achieved an accuracy of 93.0% and 84.1%, significantly higher than that of morphology (88.4% and 79.5%), ADC value (74.4% and 68.2%), and spectroscopy (81.4% and 68.2%; P < 0.05 for all). The performances of the scoring were superior to the morphology in the training dataset (areas under the receiver operating characteristics curve [AUC] = 0.95 vs. 0.89; P = 0.046), but not in the validation dataset (AUC = 0.90 vs. 0.85; P = 0.289). MDS score has potentials to improve distinguishing adenocarcinomas of cervical or endometrial origin, and warrants large-scale studies for further validation. 1 Technical Efficacy: Stage 3 J. Magn. Reson. Imaging 2018;47:1654-1666.

A color camera optical diagnostic imaging system has been developed for the measurement of temperature and velocity of individual in-flight particles in thermal plasma spray, based on the principle of particle streak velocimetry and two-color thermometry. Radiation spectral analysis of the characteristic behavior of a particle-laden plasma plume was performed to identify an optimal spectral range for accurate temperature measurements over which the powder particles and the interfering thermal plasma gas are discernable. Calibration of the thermal imaging system was carried out using a blackbody furnace with uncertainties less than 1.8%. Extensive experimental measurements were taken with the optical imaging system to obtain the temperature and velocity distributions of YSZ particles in practical plasma spraying applications. The uncertainties of the measured temperature and velocity with a color camera system mostly fluctuate around ±10% in comparison with data obtained from a commercially available measuring system.

Distributed power networks have a large number of photovoltaic power sources. The bidirection of power flow, different transient control strategies, and installation locations make the transient characteristics highly complex and unpredictable. The vast network of the distribution system makes it almost impossible to predict the electrical quantities of each branch. Reasonable aggregation modeling of the distribution network can greatly simplify the network topology, facilitating transient control and the setting of relay protection settings. An aggregated equivalent modeling method based on the LSTM neural network for distributed PV fault external characteristics is proposed. This method equates the complex distribution network to a highly nonlinear but controllable current source. The method can output the I–V curves of equivalent PV system parallel points under any output power and is able to predict the fault characteristics of the equivalent system after a voltage drop at the parallel point. Compared to traditional mechanistic modeling, this method does not require specific modeling of complex physical systems and is able to accurately map the strong nonlinear inputs and outputs of distribution networks. The established LSTM model first uses a one-dimensional convolutional layer for feature extraction of the PV power coefficients (input), and then two hidden layers are utilized to process the sequence data; the vectors are mapped into a sequence of external characteristic curves (output) in a fully connected layer. A typical distribution network is built based on the traditional PV power model, and a large number of different output combinations are selected for simulation to provide an effective training set and validation set data for LSTM model training. By using the training set data, the weights and offset coefficients of each layer of the LSTM are continuously optimized until the model with the smallest overall error is obtained, which is the optimal model. Finally, the optimal model is utilized to establish an equivalent distribution network system, different degrees of voltage drops are set up at the grid-connected points, the fault characteristics are compared with those of the complete model, and the simulation results can prove the reliability and practicality of the proposed method.