A Deep Learning-Based Hyperspectral Keypoint Representation Method and Its Application for 3D Reconstruction

Background:The accuracy of three-dimensional (3D) reconstructions from cone-beam computed tomography (CBCT) has been particularly important in dentistry, which will affect the effectiveness of diagnosis, treatment plan, and outcome in clinical practice. The aims of this study were to assess the linear, volumetric, and geometric accuracy of 3D reconstructions from CBCT and to investigate the influence of voxel size and CBCT system on the reconstructions results.Methods:Fifty teeth from 18 orthodontic patients were assigned to three groups as NewTom VG 0.15 mm group (NewTom VG; voxel size: 0.15 mm; n = 17), NewTom VG 0.30 mm group (NewTom VG; voxel size: 0.30 mm; n = 16), and VATECH DCTPRO 0.30 mm group (VATECH DCTPRO; voxel size: 0.30 mm; n = 17). The 3D reconstruction models of the teeth were segmented from CBCT data manually using Mimics 18.0 (Materialise Dental, Leuven, Belgium), and the extracted teeth were scanned by 3Shape optical scanner (3Shape A/S, Denmark). Linear and volumetric deviations were separately assessed by comparing the length and volume of the 3D reconstruction model with physical measurement by paired t-test. Geometric deviations were assessed by the root mean square value of the imposed 3D reconstruction and optical models by one-sample t-test. To assess the influence of voxel size and CBCT system on 3D reconstruction, analysis of variance (ANOVA) was used (α = 0.05).Results:The linear, volumetric, and geometric deviations were −0.03 ± 0.48 mm, −5.4 ± 2.8%, and 0.117 ± 0.018 mm for NewTom VG 0.15 mm group; −0.45 ± 0.42 mm, −4.5 ± 3.4%, and 0.116 ± 0.014 mm for NewTom VG 0.30 mm group; and −0.93 ± 0.40 mm, −4.8 ± 5.1%, and 0.194 ± 0.117 mm for VATECH DCTPRO 0.30 mm group, respectively. There were statistically significant differences between groups in terms of linear measurement (P < 0.001), but no significant difference in terms of volumetric measurement (P = 0.774). No statistically significant difference were found on geometric measurement between NewTom VG 0.15 mm and NewTom VG 0.30 mm groups (P = 0.999) while a significant difference was found between VATECH DCTPRO 0.30 mm and NewTom VG 0.30 mm groups (P = 0.006).Conclusions:The 3D reconstruction from CBCT data can achieve a high linear, volumetric, and geometric accuracy. Increasing voxel resolution from 0.30 to 0.15 mm does not result in increased accuracy of 3D tooth reconstruction while different systems can affect the accuracy.

Face recognition is still a challenging problem as there is a high possibility that the differences existing within a person or subject can exceed the differences present between different persons. Most of the current research work in the field of biometric face recognition has dealt with (Red-GreenBlue) RGB images. However, Hyper Spectral Imaging (HSI), introduces the spectral dimension for improved discrimination and leads to building a more efficient face recognition system. The spectral dimension adds more intricate details to the image. As a result hyper spectral imaging helps in improving face recognition accuracy. The faces are captured at varying spectral wavelengths of the electromagnetic spectra. Hyper spectral imaging often increases facial discrimination by capturing more biometric measurements and thus revealing information that is not revealed by the commonly used RGB images. The proposed methodology for face recognition using hyper spectral imaging includes performing band selection and band fusion on the hyper spectral face cubes and then classifying them using 3 Dimensional Convolution Neural Networks (3D-CNN). The classification using 3D-CNN gave a promising accuracy of 97.3%.

The accurate identification of melon seed varieties is essential for improving seed purity and the overall quality of melon production. In this study, hyperspectral imaging was used to identify six varieties of melon seeds. Both hyperspectral images and RGB images were generated during hyperspectral image acquisition. The spectral features of seeds were extracted from the hyperspectral images. The image features of the corresponding seeds were manually extracted from the RGB images. Five different datasets were formed using the spectral features and RGB images of the seeds, including seed spectral features, manually extracted seed image features, seed images, the fusion of seed spectral features with manually extracted features, and the fusion of seed spectral features with seed images. Logistic Regression (LR), Support Vector Classification (SVC), and Extreme Gradient Boosting (XGBoost) were used to establish classification models using spectral features and the manually extracted image features. Convolutional Neural Network (CNN) models were established using the five datasets. The results indicated that the CNN models achieved good performance in all five datasets, with classification accuracies exceeding 90% for the training, validation, and test sets. Also, CNN using the fused datasets obtained optimal performance, achieving classification accuracies exceeding 97% for the training, validation, and test sets. The results indicated that both spectral features and image features can be used to identify the six varieties of melon seeds, and their fusion of spectral features and image features can improve classification performance. These findings provide an alternative approach for melon seed variety identification, which can also be extended to other seed types.

Tristimulus display of the abundant information contained in a hyper spectral image is a challenging task. Previous visualization approaches focused on preserving as much information as possible in the reduced spectral space, but ended up with displaying hyper spectral images as false color images, which contradicts with human experience and expectation. This paper proposes a new framework to tackle this problem. It is based on the fusion of a hyper spectral image and a high-resolution color image via manifold alignment technique. Manifold learning is an important tool for dimension reduction. Manifold alignment projects a pair of two data sets into a common embedding space so that the pairs of corresponding points in these two data sets are pair wise aligned in this new space. Hyper spectral image and high-resolution color image have strong complementary properties due to the high spectral resolution in the former and the high spatial resolution in the latter. The embedding space produced by manifold alignment bridges a gap between the high dimensional spectral space of hyper spectral image and RGB space of color image, making it possible to transfer the natural color and spatial information of a high-resolution color image to a hyper spectral image to generate a visualized image with natural color distribution and finer details.

Hyperspectral image (HSI) has shown promising results in many fields because of its high spectral resolution. However, redundancy and noise in spectral dimension seriously affect the classification of HSI. For this reason, many popular dimensionality reduction (DR) methods are proposed to solve the problem. The local discriminant embedding (LDE) as an effective non-linear method for DR can be more discriminative by constructing two neighborhood graphs. However, HSI is very easy influenced by noise, and the LDE algorithm based on K nearest neighborhood is highly susceptible to interference from extreme point, which may lead to inaccurate graph construction and poor performance of classification. To overcome the problem and retain the advantages of LDE, a modified local discriminant embedding (MLDE) is firstly applied on HSI by constructing neighborhood graphs on a new spectral feature space instead of the original space. We use variance to characterize the pixels similarity of the same class and use covariance to characterize the separation of different classes of pixels. The combination of variance and covariance makes pixels in the same class to be closer and makes greater separation of pixels from different classes, which enhances classification performance of HSI. The way of representing data by using variance and covariance can attenuate the effects of noise. The Log-Euclidean metric is used to capture the similarity between spectral vectors, which can provide a more accurate similarity evaluation than euclidean distance. The experimental results of two hyperspectral datasets demonstrate the effectiveness of our proposed MLDE method.

Three-dimensional (3D) reconstruction models are useful for many situations in maintenance and decommissioning work at nuclear power plants (NPPs). For construction of 3D reconstruction models, it is desirable that they be constructed from images obtained by scanning work sites with a camera. Nevertheless, it is difficult for users to scan with being conscious of the appropriate scanning techniques and unscanned areas. Users who have no knowledge about scanning with a camera therefore have difficulty scanning work sites and capturing enough images to construct accurate 3D reconstruction models of them. In this study, we aim to develop a scanning support system using augmented reality (AR). This system detects and visualizes the current scanning status to remind and encourage users to scan as few unscanned areas as possible. For this study, we evaluated the scanning support system effectiveness by comparing the performance of developed system and non-support system. Results demonstrated that the developed system could scan larger areas and that it could capture more useful images. We also conducted an evaluation experiment at an actual NPP work site to ascertain the effectiveness, even at an actual NPP work site. Results showed that the developed system can scan larger areas, even at an actual work site.

Hyperspectral images are of high spectral resolution and have been widely used in many applications, but the imaging process to achieve high spectral resolution is at the expense of spatial resolution. This paper aims to construct a high-spatial-resolution hyperspectral (HHS) image from a high-spatial-resolution RGB image, by proposing a novel class-based spectral super-resolution method. With the help of a set of RGB and HHS image-pairs, our proposed method learns nonlinear spectral mappings between RGB and HHS image-pairs using class-based back propagation neural networks (BPNNs). In the training stage, unsupervised clustering is used to divide an RGB image into several classes according to spectral correlation, and the spectrum-pairs from the classified RGB images and the corresponding HHS images are used to train the BPNNs, to establish the nonlinear spectral mapping for each class. In the spectral super-resolution stage, a supervised classification is used to classify the given RGB image into the classes determined during the training stage, and the final HHS image is reconstructed from the classified given RGB image using the trained BPNNs. Comparisons on three standard datasets, ICVL, CAVE and NUS, demonstrate that, our proposed method achieves a better spectral super-resolution quality than related state-of-the-art methods.

Fusing a low-resolution hyperspectral image with the corresponding high-resolution RGB image to obtain a high-resolution hyperspectral image is usually solved as an optimization problem with prior-knowledge such as sparsity representation and spectral physical properties as constraints, which have limited applicability. Deep convolutional neural network extracts more comprehensive features and is proved to be effective in upsampling RGB images. However, directly applying CNNs to upsample either the spatial or spectral dimension alone may not produce pleasing results due to the neglect of complementary information from both low resolution hyper spectral and high resolution RGB images. This paper proposes two types of novel CNN architectures to take advantages of spatial and spectral fusion for hyperspectral image superresolution. Experiment results on benchmark datasets validate that the proposed spatial and spectral fusion CNNs outperforms the state-of-the-art methods and baseline CNN architectures in both quantitative values and visual qualities.

3D human face model reconstruction is essential to the generation of facial animations that is widely used in the field of virtual reality (VR). The main issues of 3D facial model reconstruction based on images by vision technologies are in twofold: one is to select and match the corresponding features of face from two images with minimal interaction and the other is to generate the realistic-looking human face model. In this paper, a new algorithm for realistic-looking face reconstruction is presented based on stereo vision. Firstly, a pattern is printed and attached to a planar surface for camera calibration, and corners generation and corners matching between two images are performed by integrating modified image pyramid Lucas-Kanade (PLK) algorithm and local adjustment algorithm, and then 3D coordinates of corners are obtained by 3D reconstruction. Individual face model is generated by the deformation of general 3D model and interpolation of the features. Finally, realistic-looking human face model is obtained after texture mapping and eyes modeling. In addition, some application examples in the field of VR are given. Experimental result shows that the proposed algorithm is robust and the 3D model is photo-realistic.

Spectroradiometric imaging is based on the fact that all materials display a specific spectral signature when light impinges on them. That is, they reflect, absorb, and emit electromagnetic waves in a particular pattern depending on the wavelength, leading to a spectrum for the material that can be used to classify it. An imaging spectroradiometer captures images where the spectral information of every pixel is collected for a set of contiguous discrete spectral bands. It is customary to classify these instruments as ultraspectral, hyperspectral, multispectral, and panchromatic imagers. Typically, ultraspectral data sets are made up of thousands of spectral bands, while hyperspectral sets have hundreds, multispectral tens, and panchromatic sets just a few. Thus, each ultraspectral or hyperspectral image contains large amounts of data which can be viewed as a cube with two spatial and one spectral dimension. Regardless of the application of the images, the analysis methods employed must deal with large quantities of data efficiently [1]. Originally, imaging spectroradiometers were used as airborne or spaceborne remote sensing instruments. During the last decade, spectral imaging has become a common advanced tool for remote sensing of the surface of Earth. It is being used to collect crucial data about many activities of economic or scientific relevance [2] as, for instance, mineral exploitation [3], agriculture and forestry use [4], environmental pollution [5], ecosystem studies [6], assessment of natural resources [7], water resource management [8], monitoring of coastal zones [9] and many others. For all these remote sensing applications the hyperspectral images obtained usually contain mixtures of spectra in every pixel due to the poor spatial resolution of the images when taken a large distance from the target. Thus a single pixel could typically correspond to one hundred or more square meters of land. As a result, most analysis methods developed had as their main objective to provide the segmentation of the images in terms of the ratio of endmembers present in every pixel to improve the spatial discrimination of these systems when analyzing different types of land covers [10].

In most hyperspectral super-resolution (HSR) methods, which are techniques used to improve the resolution of hyperspectral images (HSIs), the HSI and the target RGB image are assumed to have identical fields of view. However, because implementing these identical fields of view is difficult in practical applications, in this paper, we propose a HSR method that is applicable when an HSI and a target RGB image have different spatial information. The proposed HSR method first creates a low-resolution RGB image from a given HSI. Next, a histogram matching is performed on a high-resolution RGB image and a low-resolution RGB image obtained from an HSI. Finally, the proposed method optimizes endmember abundance of the high-resolution HSI towards the histogram-matched high-resolution RGB image. The entire procedure is evaluated using an open HSI dataset, the Harvard dataset, by adding spatial mismatch to the dataset. The spatial mismatch is implemented by shear transformation and cutting off the upper and left sides of the target RGB image. The proposed method achieved a lower error rate across the entire dataset, confirming its capability for super-resolution using images that have different fields of view.

The purpose of this study is to develop hyperspectral imaging (HSI) for automatic detection of head and neck cancer cells on histologic slides. A compact hyperspectral microscopic system is developed in this study. Histologic slides from 15 patients with squamous cell carcinoma (SCC) of the larynx and hypopharynx are imaged with the system. The proposed nuclei segmentation method based on principle component analysis (PCA) can extract most nuclei in the hyperspectral image without extracting other sub-cellular components. Both spectra-based support vector machine (SVM) and patch-based convolutional neural network (CNN) are used for nuclei classification. CNNs were trained with both hyperspectral images and pseudo RGB images of extracted nuclei, in order to evaluate the usefulness of extra information provided by hyperspectral imaging. The average accuracy of spectra-based SVM classification is 68%. The average AUC and average accuracy of the HSI patch-based CNN classification is 0.94 and 82.4%, respectively. The hyperspectral microscopic imaging and classification methods provide an automatic tool to aid pathologists in detecting SCC on histologic slides.

Malignant glioma resection is often the first line of treatment in neuro-oncology. During glioma surgery, the discrimination of tumor's edges can be challenging at the infiltration zone, even by using surgical adjuncts such as fluorescence guidance (e.g., with 5-aminolevulinic acid). Challenging cases in which there is no visible fluorescence include lower-grade gliomas, tumor cells infiltrating beyond the margin as visualized on pre- and/or intraoperative MRI, and even some high-grade tumors. One field of research aiming to address this problem involves inspecting in detail the light emission spectra from different tissues (e.g., tumor vs. normal brain vs. brain parenchyma infiltrated by tumor cells). Hyperspectral imaging measures the emission spectrum at every image pixel level, thus combining spatial and spectral information. Assuming that different tissue types have different "spectral footprints," eventually related to higher or lower abundances of fluorescent dyes or auto-fluorescing molecules, the tissue can then be segmented according to type, providing surgeons a detailed spatial map of what they see. However, processing from raw hyperspectral data cubes to maps or overlays of tissue labels and potentially further molecular information is complex. This chapter will explore some of the classical methods for the various steps of this process and examine how they can be improved with machine learning approaches. While preliminary work on machine learning in hyperspectral imaging has had relatively limited success in brain tumor surgery, more recent research combines this with fluorescence to obtain promising results. In particular, this chapter describes a pipeline that isolates biopsies in ex vivo hyperspectral fluorescence images for efficient labeling, extracts all the relevant emission spectra, preprocesses them to correct for various optical properties, and determines the abundance of fluorophores in each pixel, which correspond directly with the presence of cancerous tissue. Each step contains a combination of classical and deep learning-based methods. Furthermore, the fluorophore abundances are then used in four machine learning models to classify tumor type, WHO grade, margin tissue type, and isocitrate dehydrogenase (IDH) mutation status in brain tumors. The classifiers achieved average test accuracies of 87%, 96.1%, 86%, and 93%, respectively, thus greatly outperforming prior work both with and without fluorescence. This field is new, but these early results show great promise for the feasibility of data-driven hyperspectral imaging for intraoperative classification of brain tumors during fluorescence-guided surgery.

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

Imaging spectroscopy in the remote sensing is an ever emerging platform that has offered the hyperspectral imaging (HSI) which delivers the Earth’s object information in hundreds of bands. HSI integrates conventional imaging with spectroscopy to get rich spectral and spatial features of the object. However, the challenges associated with HSI are its huge dimensionality and data redundancy that requests huge space, complex computations, and lengthier processing time. Therefore, this study aims to find the optimal bands to characterize the roof surfaces using supervised classifiers. To deal with high dimensionality of hyperspectral data, this study assesses the band selection method over data transformation methods. This study provides the comparison between data reduction methods and used classifiers. The height information from LiDAR was used to characterize urban roofs above the height of 2.5 meters. The optimal bands were investigated using supervised classifiers such as artificial neural network (ANN), support vector machine (SVM), and spectral angle mapper (SAM) by comparing accuracies. The classification result shows that ANN and SVM classifiers outperform whereas SAM performed poorly in roof characterization. The band selection method worked efficiently than the transformation methods. The classification algorithm successfully identifies the optimum bands with significant accuracy.