Color Management within a Spectral Image Visualization Tool

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

Hyperspectral imaging (HSI) shows great promise for the detection and classification of several diseases, particularly in the fields of optical biopsy as applied to oncology, and functional retinal imaging in ophthalmology. In this paper, we discuss the application of HSI to the detection of retinal diseases and technological solutions that address some of the fundamental difficulties of spectral imaging within the eye. HSI of the retina offers a route to non-invasively deduce biochemical and metabolic processes within the retina. For example it shows promise for the mapping of retinal blood perfusion using spectral signatures of oxygenated and deoxygenated hemoglobin. Compared with other techniques using just a few spectral measurements, it offers improved classification in the presence of spectral cross-contamination by pigments and other components within the retina. There are potential applications for this imaging technique in the investigation and treatment of the eye complications of diabetes, and other diseases involving disturbances to the retinal, or optic-nerve-head circulation. It is well known that high-performance HSI requires high signal-to-noise ratios (SNR) whereas the application of any imaging technique within the eye must cope with the twin limitations of the small numerical aperture provided by the entrance pupil to the eye and the limit on the radiant power at the retina. We advocate the use of spectrally-multiplexed spectral imaging techniques (the traditional filter wheel is a traditional example). These approaches enable a flexible approach to spectral imaging, with wider spectral range, higher SNRs and lower light intensity at the retina than could be achieved using a Fourier-transform (FT) approach. We report the use of spectral imaging to provide calibrated spectral albedo images of healthy and diseased retinas and the use of this data for screening purposes. These images clearly demonstrate the ability to distinguish between oxygenated and deoxygenated hemoglobin using spectral imaging and this shows promise for the early detection of various retinopathies.

Snapshot Compressive Spectral Imaging Systems (SCSI) compress the scenes by capturing 2D projections of the encoded underlying signals. A decoder, trained with pre-acquired datasets, reconstructs the spectral images. SCSI systems based on diffractive optical elements (DOE) provide a small form factor and the single DOE can be optimized in an end-to-end manner. Since the spectral image is highly compressed in a SCSI system based on a single DOE, the quality of image reconstruction can be insufficient for diverse spectral imaging applications. This work proposes a multishot spectral imaging system employing a double-phase encoding with a double DOE architecture (DoDo), to improve the spectral reconstruction performance. The first DOE is fixed and provides the benefits of the diffractive optical systems. The second DOE provides the variable encoding of the multishot architectures. The work presents a differentiable mathematical model for the multishot DoDo system and optimizes the parameters of the DoDo architecture in an end-to-end manner. The proposed system was tested using simulations and a hardware prototype. To obtain a low-cost system, the implementation uses a deformable mirror for the variable DOE. The proposed DoDo system shows an improvement of up to 4dB in PSNR in the reconstructed spectral images compared with the single DOE system.

In this work, we propose a shared-modality multi-bandpass filtering (SMF) method for plug-and-play multispectral depth imaging (MDI). Unlike traditional multispectral filter array (MFA) cameras that require precise pixel-level alignment of spectral filters, our approach leverages a low-cost planar multi-bandpass (MBP) filter, enabling flexible and customizable multispectral imaging through simple real-time linear computation. A key advantage of our method is that, compared to conventional parallel systems, which rely on cross-modal registration to align spectral and depth images, robust shared-modal registration is achieved by integrating batch-fabricated MBPs across parallel cameras with a common passband, while also significantly reducing system volume. As a proof of concept, we developed a prototype system comprising two SMF cameras, capable of real-time 7-band spectral and depth imaging. The system achieves a full width at half maximum (FWHM) of 10-15 nm in spectral imaging, 1600 × 1300 spatial resolution, and a depth accuracy of 1.75 mm at 30 fps. Its capability was further demonstrated in an edge-computing autonomous driving scenario, validating the system's practical effectiveness.

LADMM-Net: An unrolled deep network for spectral image fusion from compressive data

Chapter 8 Spectral imaging and its use in the measurement of Förster resonance energy transfer in living cells

Pacific Advanced Technology (PAT) has developed an infrared hyperspectral camera based on diffractive optic arrays. This approach to hyperspectral imaging has been demonstrated in all three infrared bands SWIR, MWIR and LWIR. The hyperspectral optical system has been integrated into the cold-shield of the sensor enabling the small size and weight of this infrared hyperspectral sensor. This new and innovative approach to an infrared hyperspectral imaging spectrometer uses micro-optics that are made up of an area array of diffractive optical elements where each element is tuned to image a different spectral region on a common focal plane array. The lenslet array is embedded in the cold-shield of the sensor and actuated with a miniature piezo-electric motor. This approach enables rapid infrared spectral imaging with multiple spectral images collected and processed simultaneously each frame of the camera. This paper will present our optical mechanical design approach which results in an infrared hyper-spectral imaging system that is small enough for a payload on a small satellite, mini-UAV, commercial quadcopter or man portable. Also, an application of how this spectral imaging technology can easily be used to quantify the mass and volume flow rates of hydrocarbon gases. The diffractive optical elements used in the lenslet array are blazed gratings where each lenslet is tuned for a different spectral bandpass. The lenslets are configured in an area array placed a few millimeters above the focal plane and embedded in the cold-shield to reduce the background signal normally associated with the optics. The detector array is divided into sub-images covered by each lenslet. We have developed various systems using a different number of lenslets in the area array. Depending on the size of the focal plane and the diameter of the lenslet array will determine the number of simultaneous different spectral images collected each frame of the camera. A 2 x 2 lenslet array will image four different spectral images of the scene each frame and when coupled with a 512 x 512 focal plane array will give spatial resolution of 256 x 256 pixel each spectral image. Another system that we developed uses a 4 x 4 lenslet array on a 1024 x 1024 pixel element focal plane array which gives 16 spectral images of 256 x 256 pixel resolution each frame. This system spans the SWIR and MWIR bands with a single optical array and focal plane array.

Hyperspectral and multispectral imaging for evaluating food safety and quality

Spectral imaging has received a great deal of attention recently. Spectral reflectance observed from object surfaces provides crucial information in computer vision and image analysis which include the essential problems of feature detection, image segmentation, and material classification. The estimation of spectral reflectance is affected by several illumination factors such as shading, gloss, and specular highlight. The spectral invariant representations for dielectric materials only, for these factors, are inadequate for other characteristic materials like metal. In this chapter, a spectral invariant representation is introduced for obtaining reliable spectral reflectance images. The invariant formulas for spectral images of natural objects preserve spectral information and are invariant to highlights, shading, surface geometry, and illumination intensity. As an application, a material classification method is presented based on the invariant representation, which results in reliable segmentations for natural scenes and raw circuit boards spectral images.

Spectral imaging is of interest in many applications, including wide-area airborne surveillance, remote sensing, and tissue spectroscopy. Coded aperture spectral snapshot imaging (CASSI) provides an efficient mechanism to capture a 3D spectral cube with a single shot 2D measurement. CASSI uses a focal plane array (FPA) measurement of a spectrally dispersed, aperture coded, source. The spectral cube is then attained using a compressive sensing reconstruction algorithm. In this paper, we explore a new approach referred to as random convolution snapshot spectral imaging (RCSSI). It is based on FPA measurements of spectrally dispersed coherent sources that have been randomly convoluted by a spatial light modulator. The new method, based on the theory of compressive sensing via random convolutions, is shown to outperform traditional CASSI systems in terms of PSNR spectral image cube reconstructions.

To demonstrate the feasibility of low-dose photon-counting tomosynthesis in combination with a contrast agent (contrast material-enhanced tomographic mammography) for the differentiation of breast cancer. All studies were approved by the institutional review board, and all patients provided written informed consent. A phantom model with wells of iodinated contrast material (3 mg of iodine per milliliter) 1, 2, 5, 10, and 15 mm in diameter was assessed. Nine patients with malignant lesions and one with a high-risk lesion (atypical papilloma) were included (all women; mean age, 60.7 years). A multislit photon-counting tomosynthesis system was utilized (spectral imaging) to produce both low- and high-energy tomographic data (below and above the k edge of iodine, respectively) in a single scan, which allowed for dual-energy visualization of iodine. Images were obtained prior to contrast material administration and 120 and 480 seconds after contrast material administration. Four readers independently assessed the images along with conventional mammograms, ultrasonographic images, and magnetic resonance images. Glandular dose was estimated. Contrast agent was visible in the phantom model with simulated spherical tumor diameters as small as 5 mm. The average glandular dose was measured as 0.42 mGy per complete spectral imaging tomosynthesis scan of one breast. Because there were three time points (prior to contrast medium administration and 120 and 480 seconds after contrast medium administration), this resulted in a total dose of 1.26 mGy for the whole procedure in the breast with the abnormality. Seven of 10 cases were categorized as Breast Imaging Reporting and Data System score of 4 or higher by all four readers when reviewing spectral images in combination with mammograms. One lesion near the chest wall was not captured on the spectral image because of a positioning problem. The use of contrast-enhanced tomographic mammography has been demonstrated successfully in patients with promising diagnostic benefit. Further studies are necessary to fully assess diagnostic sensitivity and specificity.

We combined a homebuilt multiphoton microscope and a prism-CCD based spectrograph to develop a spectral imaging system capable of imaging deep into live tissues. The spectral images originate from the two-photon autofluorescence of the tissue and second harmonic signal from the collagen fibers. A highly penetrating near-infrared light is used to excite the endogenous fluorophores via multiphoton excitation enabling us to produce high quality images deep into the tissue. We were able to produce 100-channel (330 nm to 600 nm) autofluorescence spectral images of live skin tissues in less than 2 minutes for each xy-section. The spectral images rendered in RGB (real) colors showed green hair shafts, blue cells, and purple collagen. Analysis on the optical signal degradation with increasing depth of the collagen second-harmonic signal showed 1) exponential decay behavior of the intensity and 2) linear broadening of the spectrum. This spectral imaging system is a promising tool for both in biological applications and biomedical applications such as optical biopsy.

Temporal clustering of minced meat by RGB- and spectral imaging

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.