Learning from weakly representative data and applications in spectral image analysis

In recent years there has been an increasing interest in compressive imaging devices that capture spectral images with high spatial and spectral resolution with as few as a single snapshot. Nonetheless, there exists an intrinsic trade-off between spatial and spectral resolution which degrades one at the expense of the other. To alleviate this, state-of-art systems have relied on multiple snapshots attained via dynamically programmable spatial light modulators (SLM), or on side information from a second sensor attained via beam-splitters or stereo setups, which suffer from occlusion and demand registration tasks. This work proposes a compressive snapshot spectral imaging architecture that builds on the ideas of side information, but relies on a single image sensor. Specifically, we propose to split the incoming light into two paths, such as in a conventional interferometer, with one path dispersing the coded light, while the other just relaying it, so as to multiplex a traditional coded-and-dispersed measurement with a modulated grayscale version of the 3D cube in the image sensor. We have coined the latter, virtual side information. The proposed optical system is versatile in the sense that it can move from a fully-encoding system, to a spatially modulated grayscale camera, by switching/blocking the splitting response of the beam-splitters. Furthermore, we propose a computational algorithm to recover the underlying spectral cube exploiting the high spatial resolution provided by the virtual side information. We demonstrate through simulations from over fifty spectral images, and via an experimental proof-of-concept implementation, that the proposed imaging system together with the computational algorithm represents an efficient alternative to acquire spectral images without relying on additional sensors.

To evaluate the effect of varying spectral resolution on image quality of high spectral and spatial resolution (HiSS) images. Eight women with suspicious breast lesions and six healthy volunteers were scanned using echo-planar spectroscopic imaging (EPSI) at 1.5 Tesla with 0.75- to 1-mm in-plane resolution and 2.3- to 2.6-Hz spectral resolution. Time domain data were truncated to obtain proton spectra in each voxel with varying (2.6-83.3 Hz) resolution. Images with intensity proportional to water signal peak heights were synthesized. Changes in water signal line shape following contrast injection were analyzed. Fat suppression is optimized at approximately 10-Hz spectral resolution and is significantly improved by removal of wings of the fat resonance. This was accomplished by subtracting a Lorentzian fit to the fat resonance from the proton spectrum. The water resonance is often inhomogeneously broadened, and very high spectral resolution is necessary to resolve individual components. High spectral resolution is required for optimal contrast in anatomic features with very high T(2)* (e.g., within a lesion) and for detection of often subtle effects of contrast agents on water signal line shape. Despite a trade-off between the spectral resolution and signal-to-noise ratio, it is beneficial to acquire data at the highest spectral resolution currently attainable at 1.5 Tesla.

Conventional multispectral imaging systems based on bandpass filters struggle to record multispectral videos with high spatial resolutions because of their limited light efficiencies. This paper proposes a multi-aperture multispectral imaging system based on notch filters that overcomes this limitation by allowing light from most of the spectrum to pass through. Based on this imaging principle, a prototype multi-aperture multispectral imaging system comprising notch filters was built and demonstrated. Further, a dictionary learning- and total variation-based spectral super-resolution algorithm was developed to reconstruct spectral images. The simulation results obtained using public multispectral datasets showed that, compared to the dictionary learning-based spectral super-resolution algorithm, the proposed algorithm reconstructed the spectral information with a higher accuracy and removed noise, and the verification experiments confirmed the performance efficiency of the prototype system. The experimental results showed that the proposed imaging system can capture images with high spatial and spectral resolutions under low illumination conditions. The proposed algorithm improved the spectral resolution of the acquired data from 9 to 31 bands, and the average peak signal-to-noise ratio remained above 43 dB, which is 13 dB higher than those of the state-of-the-art coded aperture snapshot spectral imaging methods. Simultaneously, the frame rate of the imaging system was up to 5000 frames/s under natural daylight.

A spectral imager constructs a three-dimensional (two spatial and one spectral) image from a series of two-dimensional images. We discuss a technique for spectral imaging that multiplexes the spatial and spectral information on a staring focal plane and then demultiplexes the resulting imagery to obtain the spectral image. The spectral image consists of 100×100 spatial pixels and 25 spectral bands. The current implementation operates over the 3–5-μm band, but can easily be applied to other spectral regions. This approach to spectral imaging has high optical throughput and is robust to focal plane array nonuniformities. A hardware description, the mathematical development, and experimental results are presented.

Spectral imaging combines the advantages of spectroscopy as an analytical tool with the two-dimensional object visualization obtained by optical imaging. At present, spectral imaging systems suffer from severe scale-dependent geometric aberrations. This prevents the simultaneous realization of a wide field of view (FOV) and high resolution. Here we propose an ultra-wide-FOV high-resolution cascaded spectral imaging system (URIS), which captures an ultra-wide-FOV panchromatic image with a folding monocentric lens and obtains high-resolution spectral images after local aberration correction by the spectral imaging sub-channels. The key issue of how the position of the aperture stop affects the illumination uniformity of the sub-channels has been studied, contributing to achieving the spectral images stitching between adjacent sub-channels and improving the performance of the processed spectral image. The URIS is optimally designed with 19 spectral imaging sub-channels to cover a full FOV of about 110°. It achieves an excellent spatial resolution of 0.075 mrad and a high spectral resolution of 4 nm theoretically. An experiment is also demonstrated to verify the wide FOV and the uniformity of sub-channels of URIS, confirming the correctness of the theoretical study. URIS has great advantages in spectral imaging of large macroscopic scenes, inspiring various machine vision applications.

It has been noticed that spectral information can be used for analyzing and separating different biological tissues. However, most of the studies for spectral image acquisitions are mainly done in vitro. Usually the main restrictions for in vivo measurements are the size or the weight of the spectral camera. If the camera weights too much, the surgery microscope cannot be stabilized. If the size of the camera is too big, it will disturb the surgeon or even risk the safety of the patient. The main goal of this study was to develop an independent spectral imaging device which can be used for collecting spectral information from the neurosurgeries without any previously described restrictions. Size of the imaging system is small enough not to disturb the surgeon during the surgery. The developed spectral imaging system is used for collecting a spectral database which can be used for the future imaging systems.

Widely used MRI methods show brain morphology both in vivo and ex vivo at very high resolution. Many of these methods (e.g., T2*-weighted imaging, phase-sensitive imaging, or susceptibility-weighted imaging) are sensitive to local magnetic susceptibility gradients produced by subtle variations in tissue composition. However, the spectral resolution of commonly used methods is limited to maintain reasonable run-time combined with very high spatial resolution. Here, the authors report on data acquisition at increased spectral resolution, with 3-dimensional high spectral and spatial resolution MRI, in order to analyze subtle variations in water proton resonance frequency and lineshape that reflect local anatomy. The resulting information compliments previous studies based on T2* and resonance frequency. The proton free induction decay was sampled at high resolution and Fourier transformed to produce a high-resolution water spectrum for each image voxel in a 3D volume. Data were acquired using a multigradient echo pulse sequence (i.e., echo-planar spectroscopic imaging) with a spatial resolution of 50 × 50 × 70 μm(3) and spectral resolution of 3.5 Hz. Data were analyzed in the spectral domain, and images were produced from the various Fourier components of the water resonance. This allowed precise measurement of local variations in water resonance frequency and lineshape, at the expense of significantly increased run time (16-24 h). High contrast T2*-weighted images were produced from the peak of the water resonance (peak height image), revealing a high degree of anatomical detail, specifically in the hippocampus and cerebellum. In images produced from Fourier components of the water resonance at -7.0 Hz from the peak, the contrast between deep white matter tracts and the surrounding tissue is the reverse of the contrast in water peak height images. This indicates the presence of a shoulder in the water resonance that is not present at +7.0 Hz and may be specific to white matter anatomy. Moreover, a frequency shift of 6.76 ± 0.55 Hz was measured between the molecular and granular layers of the cerebellum. This shift is demonstrated in corresponding spectra; water peaks from voxels in the molecular and granular layers are consistently 2 bins apart (7.0 Hz, as dictated by the spectral resolution) from one another. High spectral and spatial resolution MR imaging has the potential to accurately measure the changes in the water resonance in small voxels. This information can guide optimization and interpretation of more commonly used, more rapid imaging methods that depend on image contrast produced by local susceptibility gradients. In addition, with improved sampling methods, high spectral and spatial resolution data could be acquired in reasonable run times, and used for in vivo scans to increase sensitivity to variations in local susceptibility.

Long-wave infrared (LWIR) spectral imaging plays a critical role in various applications such as gas monitoring, mineral exploration, and fire detection. Recent advancements in computational spectral imaging, powered by advanced algorithms, have enabled the acquisition of high-quality spectral images in real time, such as with the Uncooled Snapshot Infrared Spectrometer (USIRS). However, the USIRS system faces challenges, particularly a low spectral resolution and large amount of data noise, which can degrade the image quality. Deep learning has emerged as a promising solution to these challenges, as it is particularly effective at handling noisy data and has demonstrated significant success in hyperspectral imaging tasks. Nevertheless, the application of deep learning in LWIR imaging is hindered by the severe scarcity of long-wave hyperspectral image data, which limits the training of robust models. Moreover, existing networks that rely on convolutional layers or attention mechanisms struggle to effectively capture both local and global spectral correlations. To address these limitations, we propose the pixel-based Hierarchical Spectral Transformer (HST), a novel deep learning architecture that learns from publicly available single-pixel long-wave infrared spectral databases. The HST is designed to achieve a high spectral resolution for LWIR spectral image reconstruction, enhancing both the local and global contextual understanding of the spectral data. We evaluated the performance of the proposed method on both simulated and real-world LWIR data, demonstrating the robustness and effectiveness of the HST in improving the spectral resolution and mitigating noise, even with limited data.

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

Over recent years there has been a growing interest in molecular semiconductors of the oligoacene family. The interest has been spurred by the unique properties of these materials for applications for electronic devices, namely being °exible, cheap and exhibit- ing relatively high charge-carrier mobilities. Despite this interest, the nature of charge transport in the materials has remained the subject of intense debate. When these oligoacenes are used for device applications, they will often be evaporated as a ¯lm onto a substrate, since the fabrication process and the handling of these thin ¯lms is easier and cheaper than using single crystals. Besides the fundamental bulk properties, for the usage in devices it is also of high importance to know how the properties of these materials di®er at the interface with metals, since the substrates and contacts are often metallic. Therefore in this work both the vibrational and electronic properties of thin ¯lms of pentacene have been investigated as well as the intrinsic charge transport mechanism for the single-crystalline bulk material of pentacene and rubrene. The ¯rst main focus in this study was on the electronic and vibrational properties of thin layers of pentacene (ranging from 1 monolayer to 6 layers of pentacene molecules) evaporated on metal surfaces. These pentacene layers have been investigated using High Resolution Electron Energy Loss Spectroscopy (HREELS), which is a surface sensitive technique that probes both the vibrational and electronic states of the top layers of the sample by detecting the loss in energy of a beam of monochromatic electrons. The main advantages of this technique in comparison to techniques like IR and Raman spectroscopy are the surface sensitivity, the high spectral resolution (FWHM of 13 cm¡1), the broad energy range (from lattice phonons to surface plasmons and interband transitions) and the ability to measure as a function of wave vector transfer. Pentacene was grown upon Ag(111) and Au(100) surfaces, forming thin layers of var- ious thicknesses. The obtained vibrational spectra were compared to the vibrational modes for an isolated molecule calculated using DFT methods. The spectra of ¯lms of over 4 molecular layers showed contributions from all the optical active modes predicted by the calculation as is expected for a non-frustrated Van Der Waals bonded crystalline material. Furthermore the spectra showed low energetic features related to the lattice phonons. Both inter- and intramolecular modes showed little to no dispersion. The spec- tra for a monolayer deviated strongly from the vibrational structure of the multi-layers. Not only were the intermolecular modes absent, all intramolecular optical active modes with a dipole moment parallel to the interface were screened by the metal surface. In the case of a monolayer of pentacene on Ag(111) the interaction with the surface is so strong that the symmetric Ag modes became optically active as a result of charge transfer from the metallic substrate. For the intermediate regime (layers of 2 to 4 molecules) the spectra showed several contributions that cannot be explained using the calculated data on the symmetry and energy of the intramolecular modes. With HREELS not only the vibrational aspects of the thin ¯lms were studied but also the electronic transitions. The spectra for layers of more than four molecular layers could be modeled by combining calculations on the position of both excitonic and band levels and calculations of the vibronic ¯ne structure. For a monolayer of pentacene, states within the electronic gap were observed, the so-called Charge Neutrality Levels, which are substrate dependent. With increasing thickness the observed presence and position of the exciton states di®ered, thereby giving more insight in the structure of pentacene near the interface. Terahertz Time Domain Spectroscopy (THz TDS) is a technique that allows determin- ing the complex frequency dependent conductivity and is a powerful tool to investigate the charge transport mechanism in materials. THz TDS was used to study the charge trans- port as a function of temperature for single crystals of both pentacene and rubrene. For pentacene the conductivity spectra at various temperatures showed a strong absorption for the low energetic vibrational modes and thereby gave evidence for the model proposed by A. Troisi and G. Orlandi. This model describes the charge transport behaviour by one single mechanism: the large °uctuations in the intermolecular coupling of electronic states caused by thermal molecular motions. Furthermore it is shown that the charge density, and as a result the interaction between charges, plays an important role in the conductivity of these materials. In conclusion, this work shows that vibrational aspects are a key factor in under- standing the charge transport in oligoacene materials, and that good knowledge of the properties of these materials at the interface is essential, since di®erences in both struc- ture and electronic interaction with the substrate give rise to di®erent vibrational and electronic states in oligoacene materials.

Water resonance lineshapes observed in breast lesions imaged with high spectral and spatial resolution (HiSS) magnetic resonance imaging have been shown to contain diagnostically useful non-Lorentzian components. The purpose of this work is to update a previous method of breast lesion diagnosis by including phase-corrected absorption and dispersion spectra. This update includes information about the shape of the complex water resonance, which could improve the performance of a computer-aided diagnosis breast lesion classification scheme. The non-Lorentzian characteristics observed in complex breast lesion water resonance spectra are characterized by comparing a plot of the real versus imaginary components of the spectrum to that of a perfect complex Lorentzian spectrum, a "dispersion versus absorption" (DISPA) analysis technique. Distortion in the shape of the observed spectra indicates underlying physiologic changes, which have been shown to be correlated with malignancy. These spectral shape distortions in each lesion voxel are quantified by summing the deviations in DISPA radius from an ideal complex Lorentzian spectrum over all Fourier components, yielding a "total radial difference" (TRD). We limited our analysis to those voxels in each lesion with the largest TRD. The number of voxels considered was dependent on the lesion size. The TRD was used to classify voxels from 15 malignant and 8 benign lesions ([Formula: see text] voxels after voxel elimination). Lesion discrimination performance was evaluated for both the average and variance of the TRD within each lesion. Area under the receiver operating characteristic curve (ROC AUC) was used to assess both the voxel- and lesion-based discrimination methods in the task of distinguishing between malignant and benign. In the task of distinguishing voxels from malignant and benign lesions, TRD yielded an AUC of 0.89 (95% confidence interval [0.84, 0.91]). In the task of distinguishing malignant from benign lesions, the average radial difference yielded an AUC of 0.90 (95% confidence interval [0.71, 1.00]) and the variance in the radial difference yielded an AUC of 0.84 (95% confidence interval [0.61, 0.99]). We have applied the DISPA spectroscopic analysis method to HiSS data in order to identify and quantify voxels in breast lesions displaying non-Lorentzian characteristics. We have shown that a breast lesion classification scheme based on the absorption and dispersion spectral data obtained from HiSS acquisitions may outperform a similar classifier based on single off-peak component analysis, as it uses shape details of the entire spectrum instead of the magnitude at a single spectral location.

Optical frequency comb has revolutionized the approach to molecular spectroscopy. Here, dual-comb spectroscopy with two asynchronous electro-optic frequency combs tunable in the mid-infrared molecular fingerprint region will be reported for molecular sensing with sub-Doppler-limited spectral resolution and broad bandwidth (more than several THz within a single-shot measurement). Also a three-dimensional (3D) hyperspectral imaging technique based on femtosecond laser filamentation will be demonstrated. The feasibility of laser-induced filamentation for simultaneous detection and identification of multiple gas-phase species in a flame and the potential of the technique for information-rich 3D hyperspectral imaging with both high spatial and spectral resolution will be discussed.

This article presents an optimized algorithm for Nonnegative Tensor Factorization (NTF), implemented in the CUDA (Compute Uniform Device Architecture) framework, that runs on contemporary graphics processors and exploits their massive parallelism. The NTF implementation is primarily targeted for analysis of high-dimensional spectral images, including dimensionality reduction, feature extraction, and other tasks related to spectral imaging; however, the algorithm and its implementation are not limited to spectral imaging. The speedups measured on real spectral images are around 60-100{\times} compared to a traditional C implementation compiled with an optimizing compiler. Since common problems in the field of spectral imaging may take hours on a state-of-the-art CPU, the speedup achieved using a graphics card is attractive. The implementation is publicly available in the form of a dynamically linked library, including an interface to MATLAB, and thus may be of help to researchers and engineers using NTF on large problems.

Temporal clustering of minced meat by RGB- and spectral imaging

Hyperspectral imaging (HSI) shows great promise for the detection and classification of several diseases, particularly in the fields of optical biopsy as applied to oncology, and functional retinal imaging in ophthalmology. In this paper, we discuss the application of HSI to the detection of retinal diseases and technological solutions that address some of the fundamental difficulties of spectral imaging within the eye. HSI of the retina offers a route to non-invasively deduce biochemical and metabolic processes within the retina. For example it shows promise for the mapping of retinal blood perfusion using spectral signatures of oxygenated and deoxygenated hemoglobin. Compared with other techniques using just a few spectral measurements, it offers improved classification in the presence of spectral cross-contamination by pigments and other components within the retina. There are potential applications for this imaging technique in the investigation and treatment of the eye complications of diabetes, and other diseases involving disturbances to the retinal, or optic-nerve-head circulation. It is well known that high-performance HSI requires high signal-to-noise ratios (SNR) whereas the application of any imaging technique within the eye must cope with the twin limitations of the small numerical aperture provided by the entrance pupil to the eye and the limit on the radiant power at the retina. We advocate the use of spectrally-multiplexed spectral imaging techniques (the traditional filter wheel is a traditional example). These approaches enable a flexible approach to spectral imaging, with wider spectral range, higher SNRs and lower light intensity at the retina than could be achieved using a Fourier-transform (FT) approach. We report the use of spectral imaging to provide calibrated spectral albedo images of healthy and diseased retinas and the use of this data for screening purposes. These images clearly demonstrate the ability to distinguish between oxygenated and deoxygenated hemoglobin using spectral imaging and this shows promise for the early detection of various retinopathies.