Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning

Hyperspectral imagery (HSI) and light detection and ranging (LiDAR) remote sensing technologies are important ways to obtain surface information. Combining the characteristics and advantages of hyperspectral images and LiDAR DSM data, we propose a classification method for co-classification of hyperspectral images and LiDAR data in this paper. The model uses a dual-branch HSI image and LiDAR data classification method. First, the features of shallow convolution and deep convolution are merged in the spatial feature extraction process of HSI, which combines to focus on more global information. Then the space and spectrum are combined, considering the characteristics of different layers, the network has the characteristics of multiscale. Finally, the LiDAR branch introduces dilated convolution to obtain a larger receptive field and reduce information loss. The classification experimental results on Houston and Trento dataset show the superiority of our proposed method in the collaborative classification of hyperspectral images and LiDAR data.

A new framework is proposed for the fusion of hyperspectral and light detection and ranging (LiDAR) data based on the extinction profiles (EPs), local binary pattern (LBP), and kernel collaborative representation classification. Specifically, EP and LBP features are extracted from both sources. Then, the derived features of each source are classified by collaborative representation-based classifier with Tikhonov regularization (KCRT). Reconstruction residuals are fused to produce the final label assignment. Experimental results demonstrate that the proposed method outperforms the existing methods in hyperspectral and LiDAR data fusion.

The 2013 Data Fusion Contest organized by the Data Fusion Technical Committee (DFTC) of the IEEE Geoscience and Remote Sensing Society aimed at investigating the synergistic use of hyperspectral and Light Detection And Ranging (LiDAR) data. The data sets distributed to the participants during the Contest, a hyperspectral imagery and the corresponding LiDAR-derived digital surface model (DSM), were acquired by the NSF-funded Center for Airborne Laser Mapping over the University of Houston campus and its neighboring area in the summer of 2012. This paper highlights the two awarded research contributions, which investigated different approaches for the fusion of hyperspectral and LiDAR data, including a combined unsupervised and supervised classification scheme, and a graph-based method for the fusion of spectral, spatial, and elevation information.

With the increasing number of remote sensing (RS) data sources, the joint utilization of multimodal data in Earth observation tasks has become a crucial research topic. As a typical representative of RS data, hyperspectral images (HSIs) provide accurate spectral information, while rich elevation information can be obtained from light detection and ranging (LiDAR) data. However, due to the significant differences in multimodal heterogeneous features, how to efficiently fuse HSI and LiDAR data remains one of the challenges faced by existing research. In addition, the edge contour information of images is not fully considered by existing methods, which can easily lead to performance bottlenecks. Thus, a joint classification network of HSI and LiDAR data based on Mamba (HLMamba) is proposed. Specifically, a gradient joint algorithm (GJA) is first performed on LiDAR data to obtain the edge contour data of the land distribution. Subsequently, a multimodal feature extraction module (MFEM) was proposed to capture the semantic features of HSI, LiDAR, and edge contour data. Then, to efficiently fuse multimodal features, a novel deep learning (DL) framework called Mamba, was introduced, and a multimodal Mamba fusion module (MMFM) was constructed. By efficiently modeling the long-distance dependencies of multimodal sequences, the MMFM can better explore the internal features of multimodal data and the interrelationships between modalities, thereby enhancing fusion performance. Finally, to validate the effectiveness of HLMamba, a series of experiments were conducted on three common HSI and LiDAR datasets. The results indicate that HLMamba has superior classification performance compared to other state-of-the-art DL methods. The source code of the proposed method will be available publicly at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Dilingliao/HLMamba</uri>.

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

Recently, the fusion of hyperspectral and light detection and ranging (LiDAR) data has obtained a great attention in the remote sensing community. In this paper, we propose a new feature fusion framework using deep neural network (DNN). The proposed framework employs a novel 3D convolutional neural network (CNN) to extract the spectral-spatial features of hyperspectral data, a deep 2D CNN to extract the elevation features of LiDAR data, and then a fully connected deep neural network to fuse the extracted features in the previous CNNs. Through the aforementioned three deep networks, one can extract the discriminant and invariant features of hyperspectral and LiDAR data. At last, logistic regression is used to produce the final classification results. The experimental results reveal that the proposed deep fusion model provides competitive results. Furthermore, the proposed deep fusion idea opens a new window for future research.

Recent growth in unmanned aerial vehicle (UAV) technology have promoted the detailed mapping of individual tree species. However, the in-depth mining and comprehending of the significance of features derived from high-resolution UAV data for tree species discrimination remains a difficult task. In this study, a state-of-the-art approach combining UAV-borne light detection and ranging (LiDAR) and hyperspectral was used to classify 11 common tree species in a typical natural secondary forest in Northeast China. First, comprehensive relevant structural and spectral features were extracted. Then, the most valuable feature sets were selected by using a hybrid approach combining correlation-based feature selection with the optimized recursive feature elimination algorithm. The random forest algorithm was used to assess feature importance and perform the classification. Finally, the robustness of features derived from point clouds with different structures and hyperspectral images with different spatial resolutions was tested. Our results showed that the best classification accuracy was obtained by combining LiDAR and hyperspectral data (75.7%) compared to that based on LiDAR (60.0%) and hyperspectral (64.8%) data alone. The mean intensity of single returns and the visible atmospherically resistant index for red-edge band were the most influential LiDAR and hyperspectral derived features, respectively. The selected features were robust in point clouds with a density not lower than 5% (~5 pts/m2) and a resolution not lower than 0.3 m in hyperspectral data. Although canopy surface features were slightly different from original LiDAR features, canopy surface information was also important for tree species classification. This study proved the capabilities of UAV-borne LiDAR and hyperspectral data in natural secondary forest tree species discrimination and the potential for this approach to be transferable to other study areas.

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data is widely used for land cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image differences, and the dimensions and feature distributions of HSI and LiDAR are highly dissimilar. This makes it challenging to represent and correlate semantic information from multimodal data. Current methods for classifying pixel-by-pixel features, which rely on cascaded or attention-based fusion, cannot effectively use multimodal features. To achieve accurate classification results, extracting and fusing similar high-order semantic information and complementary discriminative information contained in multimodal data is vital. In this paper, we propose a cross-modal semantic enhancement network (CMSE) for multimodal semantic information mining and fusion. Our proposed CMSE framework extracts features from the image on multiple scales, capturing more representative local sparse features with different sizes of convolution kernels. To represent high-level semantic features related to land cover, we establish a Gaussian-weighted matrix and semantically transform the spatial and spectral features of distinct branches. Finally, we build a multi-level residual fusion module to incrementally fuse spectral features from HSI and elevation features from LiDAR. Additionally, we introduce a cross-modal semantically constrained loss to guide multimodal semantic feature alignment. We evaluate our approach on three multimodal remote sensing datasets, namely the Houston2013, Trento, and MUUFL datasets. The experimental results demonstrate that our proposed CMSE model achieves superior performance in terms of accuracy and robustness compared to other related deep networks.

The joint use of information from hyperspectral image (HSI) data and light detection and ranging (LiDAR) data enables the effective identification and classification of complex land cover categories. Recently, many deep learning classification methods based on the combination of convolutional neural networks (CNN) and Transformer have been used in HSI and LiDAR classification task. However, existing algorithms pay little attention to the fusion of centre pixel features in multimodal data. In this article, we propose a novel centre pixel features enhancement network (CPFENet) for effective cross-modal information fusion. Specifically, this article introduces a centre pixel enhancement module (CPEM), which integrates absolute positional encoding and centre relative position encoding to enhance the expression of centre pixel features. Subsequently, the enhanced features are fed into a novel centre pixel feature cross-attention (CPFCA) module for fusion, which enables the network to focus on the centre pixel to learn spatial and spectral features. Experimental results on three public datasets demonstrate that the performance of the proposed network is better than that of previous state-of-the-art methods.

Deep learning (DL) has been garnering increasing attention in remote sensing (RS) due to its powerful data representation ability. In particular, deep models have been proven to be effective for RS data classification based on a single given modality. However, with one single modality, the ability in identifying the materials remains limited due to the lack of feature diversity. To overcome this limitation, we present a simple but effective multimodal DL baseline by following a deep encoder–decoder network architecture, EndNet for short, for the classification of hyperspectral and light detection and ranging (LiDAR) data. EndNet fuses the multimodal information by enforcing the fused features to reconstruct the multimodal input in turn. Such a reconstruction strategy is capable of better activating the neurons across modalities compared with some conventional and widely used fusion strategies, e.g., early fusion, middle fusion, and late fusion. Extensive experiments conducted on two popular hyperspectral and LiDAR data sets demonstrate the superiority and effectiveness of the proposed EndNet in comparison with several state-of-the-art baselines in the hyperspectral-LiDAR classification task. The codes will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/danfenghong/IEEE_GRSL_EndNet</uri> , contributing to the RS community.

Light Detection and Ranging (LIDAR) has become a very important tool to many environmental applications. This work proposes to use LIDAR and image data fusion to develop high-resolution thematic maps. A novel methodology is presented which starts building a matrix of statistics from spectral and spatial information by feature extraction on the available bands (RGB from images, and intensity and height from LIDAR). Then, a contextual classification is applied to generate the final map using a support vector machine (SVM) to classify every cell and the nearest neighbor (NN) rule to sequentially reclassify each cell. The results obtained by this novel method, called SVMNNS (SVM and NN Stacking), are compared with non-contextual and contextual SVMs. It is shown that SVMNNS obtains the best results when applied to real data from the Iberian peninsula.

To increase understanding of forest carbon cycles and stocks, estimates of total and component (e.g. leaf, branch and trunk) biomass at a range of scales are desirable. Focusing on mixed species forests in central south‐east Queensland, two different approaches to the retrieval of biomass from small footprint Light Detection and Ranging (LiDAR) and Compact Airborne Spectrographic Imager (CASI) hyperspectral data were developed and compared. In the first, stems were located using a LiDAR crown openness index, and each was associated with crowns delineated and identified to species using CASI data. The component biomass for individual trees was then estimated using LiDAR‐derived height and stem diameter as input to species‐specific allometric equations. When summed to give total above‐ground biomass (AGB) and aggregated to the plot level, these estimates showed a reasonable correspondence with ground (plot‐based) estimates (r 2 = 0.56, RSE = 25.3 Mg ha−1, n = 21) given the complex forest being assessed. In the second approach, a Jackknife linear regression utilizing six LiDAR strata heights and crown cover at the plot‐scale produced more robust estimates of AGB that showed a closer correspondence with plot‐scale ground data (r 2 = 0.90, RSE = 11.8 Mg ha−1, n = 31). AGB aggregated from the tree‐level and Jackknife regression plot‐based AGB estimates (for 270 plots—each of 0.25 ha) compared well for more mature homogeneous and open forests. However, at the tree level, AGB was overestimated in taller forests dominated by trees with large spreading crowns, and underestimated AGB where an understorey with a high density of stems occurred. The study demonstrated options for quantifying component biomass and AGB through integration of LiDAR and CASI data but highlighted the requirement for methods that give improved estimation of tree density (by size class distributions) and species occurrence in complex forests.

Accurate estimation of the fraction of absorbed photosynthetically active radiation (fPAR) is important for maize growth and yield estimations. Light detection and ranging (LiDAR)-derived canopy vertical structural and hyperspectral image-derived vegetation spectral information are complementary for vegetation fPAR estimation. This study explores the potential of artificial neural networks (ANNs) with two types of data to estimate maize fPAR. First, 45 metrics were derived from LiDAR data and 13 from a hyperspectral image. Then, the ANNs and stepwise multiple linear regression (SMLR) methods were used to estimate the fPAR. Finally, model validity was assessed using in-situ data. Results showed that the ANNs performed better in fPAR inversion (R 2 = 0.910, adj. R 2 = 0.921, RMSE = 0.046, RRMSE = 0.056, where R 2 is the coefficient of determination, adj. R 2 the adjusted coefficient of determination, RMSE the root mean squared error, and RRMSE the relative root mean squared error) than SMLR (R 2 = 0.638, adj. R 2 = 0.609, RMSE = 0.077, RRMSE = 0.092) and SMLR with the natural logarithm of data (R 2 = 0.855, adj. R 2 = 0.825, RMSE = 0.067, RRMSE = 0.081). This study is helpful for guiding the accurate estimation of maize fPAR using remote sensing techniques.

Civil engineering includes a wide range of applications that require three-dimensional (3D) information. With the advent of LiDAR (Light Detection And Ranging) technology, obtaining 3D terrain information accurately in a cost and time effective manner became more popular. In this paper, three case studies demonstrate the roles and applications of airborne LiDAR data including highway, transmission line and hydropower dam development projects. In first case study, the highway alignment was considerably changed after the initial design stage. LiDAR data was able to do the re-alignment survey quickly, allowing for prompt changes in design. In the next case study, a comparison is made between conventional transmission route alignment design and the design using PLS-CADD with LiDAR data. Utilizing the LiDAR data in PLSCADD allows transmission line engineers to model the construction costs of several alignments options effectively and efficiently compared to the conventional method. The last case study uses LiDAR for obtaining the necessary information for proposed dam. Using LiDAR data maximizes topological accuracy and volume measurements while minimizing survey time and costs. Furthermore, utilizing the multi-temporal LiDAR data can reveal valuable information such as topography changes. LiDAR survey technology has proven that it is the best method to capture accurate data from the earth surface which provides maximum density of topographical &amp; digital imagery information for various applications.

Airborne light detection and ranging (LIDAR) system uses in many field such as military, agriculture, geology and transportation. LIDAR data acquisition consumes several gigabytes from the storage capacity of the system. So, it's necessary to reduce the on board storage capacity as well as its transmission rate. Compression of LIDAR data means reducing the amount of data required to represent the digital data acquired by LIDAR system. This is done by removing data redundancies to achieve compression. In this paper we explain LIDAR system theory and component, compare between lossy and lossless compression techniques then we discuss LIDAR data compression techniques in order to choose a suitable one for LIDAR system that gives high compression ratio with suitable processing time. In most LIDAR data compression techniques classification of LIDAR data is an important issue as a preprocessing step to simplify the data. Using more than one compression algorithms increase the compression ratio but increase the processing time as well.