High-throughput hyperspectral infrared camera

Unlike straightforward registration problems encountered in broadband imaging, spectral imaging in fielded instruments often suffers from a combination of imaging aberrations that make spatial co-registration of the images a challenging problem. Depending on the sensor architecture, typical problems to be mitigated include differing focus, magnification, and warping between the images in the various spectral bands due to optics differences; scene shift between spectral images due to parallax; and scene shift due to temporal misregistration between the spectral images. However, typical spectral images sometimes contain scene commonalities that can be exploited in traditional ways. As a first step toward automatic spatial co-registration for spectral images, we exploit manually-selected scene commonalities to produce transformation parameters in a four-channel spectral imager. The four bands consist of two mid-wave infrared channels and two short-wave infrared channels. Each of the four bands is blurred differently due to differing focal lengths of the imaging optics, magnified differently, warped differently, and translated differently. Centroid location techniques are used on the scene commonalities in order to generate sub-pixel values for the fiducial markers used in the transformation polygons, and conclusions are drawn about the effectiveness of such techniques in spectral imaging applications.

Coded aperture snapshot spectral imager (CASSI), efficiently captures 3D spectral images by sensing 2D projections of the scene. While CASSI offers a substantial reduction in acquisition time, compared to traditional scanning optical systems, it requires a reconstruction post-processing step. Furthermore, to obtain high-quality reconstructions, an accurate propagation model is required. Notably, CASSI exhibits a variant spatio-spectral sensor response, making it difficult to acquire an accurate propagation model. To address these inherent limitations, this work proposes to learn a deep non-linear fully differentiable propagation model that can be used as a regularizer within an optimization-based reconstruction algorithm. The proposed approach trains the non-linear spatially-variant propagation model using paired compressed measurements and spectral images, by employing side information only in the calibration step. From the deep propagation model incorporation into a plug-and-play alternating direction method of multipliers framework, our proposed method outperforms traditional CASSI linear-based models. Extensive simulations and a testbed implementation validate the efficacy of the proposed methodology.

Pacific Advanced Technology (PAT) has developed an infrared hyperspectral camera based on diffractive optic arrays. This approach to hyperspectral imaging has been demonstrated in all three infrared bands SWIR, MWIR and LWIR. The hyperspectral optical system has been integrated into the cold-shield of the sensor enabling the small size and weight of this infrared hyperspectral sensor. This new and innovative approach to an infrared hyperspectral imaging spectrometer uses micro-optics that are made up of an area array of diffractive optical elements where each element is tuned to image a different spectral region on a common focal plane array. The lenslet array is embedded in the cold-shield of the sensor and actuated with a miniature piezo-electric motor. This approach enables rapid infrared spectral imaging with multiple spectral images collected and processed simultaneously each frame of the camera. This paper will present our optical mechanical design approach which results in an infrared hyper-spectral imaging system that is small enough for a payload on a small satellite, mini-UAV, commercial quadcopter or man portable. Also, an application of how this spectral imaging technology can easily be used to quantify the mass and volume flow rates of hydrocarbon gases. The diffractive optical elements used in the lenslet array are blazed gratings where each lenslet is tuned for a different spectral bandpass. The lenslets are configured in an area array placed a few millimeters above the focal plane and embedded in the cold-shield to reduce the background signal normally associated with the optics. The detector array is divided into sub-images covered by each lenslet. We have developed various systems using a different number of lenslets in the area array. Depending on the size of the focal plane and the diameter of the lenslet array will determine the number of simultaneous different spectral images collected each frame of the camera. A 2 x 2 lenslet array will image four different spectral images of the scene each frame and when coupled with a 512 x 512 focal plane array will give spatial resolution of 256 x 256 pixel each spectral image. Another system that we developed uses a 4 x 4 lenslet array on a 1024 x 1024 pixel element focal plane array which gives 16 spectral images of 256 x 256 pixel resolution each frame. This system spans the SWIR and MWIR bands with a single optical array and focal plane array.

Hyperspectral imaging (HSI) shows great promise for the detection and classification of several diseases, particularly in the fields of optical biopsy as applied to oncology, and functional retinal imaging in ophthalmology. In this paper, we discuss the application of HSI to the detection of retinal diseases and technological solutions that address some of the fundamental difficulties of spectral imaging within the eye. HSI of the retina offers a route to non-invasively deduce biochemical and metabolic processes within the retina. For example it shows promise for the mapping of retinal blood perfusion using spectral signatures of oxygenated and deoxygenated hemoglobin. Compared with other techniques using just a few spectral measurements, it offers improved classification in the presence of spectral cross-contamination by pigments and other components within the retina. There are potential applications for this imaging technique in the investigation and treatment of the eye complications of diabetes, and other diseases involving disturbances to the retinal, or optic-nerve-head circulation. It is well known that high-performance HSI requires high signal-to-noise ratios (SNR) whereas the application of any imaging technique within the eye must cope with the twin limitations of the small numerical aperture provided by the entrance pupil to the eye and the limit on the radiant power at the retina. We advocate the use of spectrally-multiplexed spectral imaging techniques (the traditional filter wheel is a traditional example). These approaches enable a flexible approach to spectral imaging, with wider spectral range, higher SNRs and lower light intensity at the retina than could be achieved using a Fourier-transform (FT) approach. We report the use of spectral imaging to provide calibrated spectral albedo images of healthy and diseased retinas and the use of this data for screening purposes. These images clearly demonstrate the ability to distinguish between oxygenated and deoxygenated hemoglobin using spectral imaging and this shows promise for the early detection of various retinopathies.

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

Hyperspectral and multispectral imaging for evaluating food safety and quality

Spectral imaging has received a great deal of attention recently. Spectral reflectance observed from object surfaces provides crucial information in computer vision and image analysis which include the essential problems of feature detection, image segmentation, and material classification. The estimation of spectral reflectance is affected by several illumination factors such as shading, gloss, and specular highlight. The spectral invariant representations for dielectric materials only, for these factors, are inadequate for other characteristic materials like metal. In this chapter, a spectral invariant representation is introduced for obtaining reliable spectral reflectance images. The invariant formulas for spectral images of natural objects preserve spectral information and are invariant to highlights, shading, surface geometry, and illumination intensity. As an application, a material classification method is presented based on the invariant representation, which results in reliable segmentations for natural scenes and raw circuit boards spectral images.

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

Spectral imaging has been extensively applied in many fields, including agriculture, environmental monitoring, biomedical diagnostics, etc. Thanks to the advances in sensor technology, spectral imaging systems nowadays provide finer and finer spectral resolution needed to characterize the spectral properties of materials. The high spectral resolution, however, raises an issue as the difference in spectral information between two adjacent wavelength bands is typically very small. As a result, much of the data in a scene seems to be redundant. However, critical information is embedded that often can be used to identify materials. This thesis aims at facilitating the analysis in spectral imaging by making use of pattern recognition techniques, on the one hand, to improve visualization, and on the other hand, to directly solve classification problems.

Snapshot Compressive Spectral Imaging Systems (SCSI) compress the scenes by capturing 2D projections of the encoded underlying signals. A decoder, trained with pre-acquired datasets, reconstructs the spectral images. SCSI systems based on diffractive optical elements (DOE) provide a small form factor and the single DOE can be optimized in an end-to-end manner. Since the spectral image is highly compressed in a SCSI system based on a single DOE, the quality of image reconstruction can be insufficient for diverse spectral imaging applications. This work proposes a multishot spectral imaging system employing a double-phase encoding with a double DOE architecture (DoDo), to improve the spectral reconstruction performance. The first DOE is fixed and provides the benefits of the diffractive optical systems. The second DOE provides the variable encoding of the multishot architectures. The work presents a differentiable mathematical model for the multishot DoDo system and optimizes the parameters of the DoDo architecture in an end-to-end manner. The proposed system was tested using simulations and a hardware prototype. To obtain a low-cost system, the implementation uses a deformable mirror for the variable DOE. The proposed DoDo system shows an improvement of up to 4dB in PSNR in the reconstructed spectral images compared with the single DOE system.

In this work, we propose a shared-modality multi-bandpass filtering (SMF) method for plug-and-play multispectral depth imaging (MDI). Unlike traditional multispectral filter array (MFA) cameras that require precise pixel-level alignment of spectral filters, our approach leverages a low-cost planar multi-bandpass (MBP) filter, enabling flexible and customizable multispectral imaging through simple real-time linear computation. A key advantage of our method is that, compared to conventional parallel systems, which rely on cross-modal registration to align spectral and depth images, robust shared-modal registration is achieved by integrating batch-fabricated MBPs across parallel cameras with a common passband, while also significantly reducing system volume. As a proof of concept, we developed a prototype system comprising two SMF cameras, capable of real-time 7-band spectral and depth imaging. The system achieves a full width at half maximum (FWHM) of 10-15 nm in spectral imaging, 1600 × 1300 spatial resolution, and a depth accuracy of 1.75 mm at 30 fps. Its capability was further demonstrated in an edge-computing autonomous driving scenario, validating the system's practical effectiveness.

LADMM-Net: An unrolled deep network for spectral image fusion from compressive data

Chapter 8 Spectral imaging and its use in the measurement of Förster resonance energy transfer in living cells

We combined a homebuilt multiphoton microscope and a prism-CCD based spectrograph to develop a spectral imaging system capable of imaging deep into live tissues. The spectral images originate from the two-photon autofluorescence of the tissue and second harmonic signal from the collagen fibers. A highly penetrating near-infrared light is used to excite the endogenous fluorophores via multiphoton excitation enabling us to produce high quality images deep into the tissue. We were able to produce 100-channel (330 nm to 600 nm) autofluorescence spectral images of live skin tissues in less than 2 minutes for each xy-section. The spectral images rendered in RGB (real) colors showed green hair shafts, blue cells, and purple collagen. Analysis on the optical signal degradation with increasing depth of the collagen second-harmonic signal showed 1) exponential decay behavior of the intensity and 2) linear broadening of the spectrum. This spectral imaging system is a promising tool for both in biological applications and biomedical applications such as optical biopsy.

It has been noticed that spectral information can be used for analyzing and separating different biological tissues. However, most of the studies for spectral image acquisitions are mainly done in vitro. Usually the main restrictions for in vivo measurements are the size or the weight of the spectral camera. If the camera weights too much, the surgery microscope cannot be stabilized. If the size of the camera is too big, it will disturb the surgeon or even risk the safety of the patient. The main goal of this study was to develop an independent spectral imaging device which can be used for collecting spectral information from the neurosurgeries without any previously described restrictions. Size of the imaging system is small enough not to disturb the surgeon during the surgery. The developed spectral imaging system is used for collecting a spectral database which can be used for the future imaging systems.