Multimodal fusion network with learnable wavelet-enhanced features for hyperspectral unmixing

With the increasing number of remote sensing (RS) data sources, the joint utilization of multimodal data in Earth observation tasks has become a crucial research topic. As a typical representative of RS data, hyperspectral images (HSIs) provide accurate spectral information, while rich elevation information can be obtained from light detection and ranging (LiDAR) data. However, due to the significant differences in multimodal heterogeneous features, how to efficiently fuse HSI and LiDAR data remains one of the challenges faced by existing research. In addition, the edge contour information of images is not fully considered by existing methods, which can easily lead to performance bottlenecks. Thus, a joint classification network of HSI and LiDAR data based on Mamba (HLMamba) is proposed. Specifically, a gradient joint algorithm (GJA) is first performed on LiDAR data to obtain the edge contour data of the land distribution. Subsequently, a multimodal feature extraction module (MFEM) was proposed to capture the semantic features of HSI, LiDAR, and edge contour data. Then, to efficiently fuse multimodal features, a novel deep learning (DL) framework called Mamba, was introduced, and a multimodal Mamba fusion module (MMFM) was constructed. By efficiently modeling the long-distance dependencies of multimodal sequences, the MMFM can better explore the internal features of multimodal data and the interrelationships between modalities, thereby enhancing fusion performance. Finally, to validate the effectiveness of HLMamba, a series of experiments were conducted on three common HSI and LiDAR datasets. The results indicate that HLMamba has superior classification performance compared to other state-of-the-art DL methods. The source code of the proposed method will be available publicly at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Dilingliao/HLMamba</uri>.

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

Accurate estimation of the fraction of absorbed photosynthetically active radiation (fPAR) is important for maize growth and yield estimations. Light detection and ranging (LiDAR)-derived canopy vertical structural and hyperspectral image-derived vegetation spectral information are complementary for vegetation fPAR estimation. This study explores the potential of artificial neural networks (ANNs) with two types of data to estimate maize fPAR. First, 45 metrics were derived from LiDAR data and 13 from a hyperspectral image. Then, the ANNs and stepwise multiple linear regression (SMLR) methods were used to estimate the fPAR. Finally, model validity was assessed using in-situ data. Results showed that the ANNs performed better in fPAR inversion (R 2 = 0.910, adj. R 2 = 0.921, RMSE = 0.046, RRMSE = 0.056, where R 2 is the coefficient of determination, adj. R 2 the adjusted coefficient of determination, RMSE the root mean squared error, and RRMSE the relative root mean squared error) than SMLR (R 2 = 0.638, adj. R 2 = 0.609, RMSE = 0.077, RRMSE = 0.092) and SMLR with the natural logarithm of data (R 2 = 0.855, adj. R 2 = 0.825, RMSE = 0.067, RRMSE = 0.081). This study is helpful for guiding the accurate estimation of maize fPAR using remote sensing techniques.

The joint classification of hyperspectral imaging (HSI) and Light Detection and Ranging (LiDAR) data holds significant importance for various practical uses, including urban mapping, mineral prospecting, and ecological observation. Achieving robust and transferable feature representations is essential to fully leverage the complementary properties of HSI and LiDAR modalities. However, existing methods are often constrained to scene-specific training and lack generalizability across datasets, limiting their discriminative power. To tackle this challenge, we introduce a new dual-phase approach for the combined classification of HSI and LiDAR data. Initially, a transformer-driven network is trained on various HSI-only datasets to extract universal spatial–spectral features. In the second stage, LiDAR data is incorporated as a task-specific prompt to adapt the model to HSI-LiDAR scenes and enable effective multimodal fusion. Through extensive testing on three benchmark datasets, our framework proves highly effective, outperforming all competing approaches.

Civil engineering includes a wide range of applications that require three-dimensional (3D) information. With the advent of LiDAR (Light Detection And Ranging) technology, obtaining 3D terrain information accurately in a cost and time effective manner became more popular. In this paper, three case studies demonstrate the roles and applications of airborne LiDAR data including highway, transmission line and hydropower dam development projects. In first case study, the highway alignment was considerably changed after the initial design stage. LiDAR data was able to do the re-alignment survey quickly, allowing for prompt changes in design. In the next case study, a comparison is made between conventional transmission route alignment design and the design using PLS-CADD with LiDAR data. Utilizing the LiDAR data in PLSCADD allows transmission line engineers to model the construction costs of several alignments options effectively and efficiently compared to the conventional method. The last case study uses LiDAR for obtaining the necessary information for proposed dam. Using LiDAR data maximizes topological accuracy and volume measurements while minimizing survey time and costs. Furthermore, utilizing the multi-temporal LiDAR data can reveal valuable information such as topography changes. LiDAR survey technology has proven that it is the best method to capture accurate data from the earth surface which provides maximum density of topographical &amp; digital imagery information for various applications.

Although the data fusion of hyperspectral images (HSI) and light detection and ranging (LiDAR) has provided significant gains for land-cover classification, it also brings technical obstacles (i.e., it is difficult to capture discriminative local and global spatial-spectral from redundant data and build interactions between heterogeneous data). In this paper, a classification network named enhanced spectral attention and adaptive spatial learning guided network (ESASNet) is proposed for the joint use of HSI and LiDAR. Specifically, first, by combining a convolutional neural network (CNN) with the transformer, adaptive spatial learning (ASL) and enhanced spectral learning (ESL) are proposed to learn the spectral-spatial features from the HSI data and the elevation features from the LiDAR data in the local and global receptive field. Second, considering the characteristics of HSI with a continuous, narrowband spectrum, ESL is designed by adding enhanced local self-attention to enhance the mining of the spectral correlations across the adjacent spectrum. Finally, a feature fusion module is proposed to ensure an efficient information exchange between HSI and LiDAR during spectral features and spatial feature fusion. Experimental evaluations on the HSI-LiDAR dataset clearly illustrate that ESASNet performs better in feature extraction than the state-of-the-art methods. The code is available at https://github.com/AirsterMode/ESASNet.

Attribute-guided feature fusion network with knowledge-inspired attention mechanism for multi-source remote sensing classification.

Recent growth in unmanned aerial vehicle (UAV) technology have promoted the detailed mapping of individual tree species. However, the in-depth mining and comprehending of the significance of features derived from high-resolution UAV data for tree species discrimination remains a difficult task. In this study, a state-of-the-art approach combining UAV-borne light detection and ranging (LiDAR) and hyperspectral was used to classify 11 common tree species in a typical natural secondary forest in Northeast China. First, comprehensive relevant structural and spectral features were extracted. Then, the most valuable feature sets were selected by using a hybrid approach combining correlation-based feature selection with the optimized recursive feature elimination algorithm. The random forest algorithm was used to assess feature importance and perform the classification. Finally, the robustness of features derived from point clouds with different structures and hyperspectral images with different spatial resolutions was tested. Our results showed that the best classification accuracy was obtained by combining LiDAR and hyperspectral data (75.7%) compared to that based on LiDAR (60.0%) and hyperspectral (64.8%) data alone. The mean intensity of single returns and the visible atmospherically resistant index for red-edge band were the most influential LiDAR and hyperspectral derived features, respectively. The selected features were robust in point clouds with a density not lower than 5% (~5 pts/m2) and a resolution not lower than 0.3 m in hyperspectral data. Although canopy surface features were slightly different from original LiDAR features, canopy surface information was also important for tree species classification. This study proved the capabilities of UAV-borne LiDAR and hyperspectral data in natural secondary forest tree species discrimination and the potential for this approach to be transferable to other study areas.

Foresters are increasingly interested in remote sensing data because they provide an overview of landscape conditions, which is impractical with field sample data alone. Light Detection and Ranging (LiDAR) provides exceptional spatial detail of forest structure, but difficulties in processing LiDAR data have limited their application beyond the research community. Another obstacle to operational use of LiDAR data has been the high cost of data collection. Our objectives in this study were to summarize, at the stand level, both LiDAR- and Landsat (satellite)-based predictions of some common structural and volume attributes and to compare the cost of obtaining such summaries with those obtained through traditional stand exams. We found that the accuracy and cost of a LiDAR-based inventory summarized at the stand level was comparable to traditional stand exams for structural attributes. However, the LiDAR data were able to provide information across a much larger area than the stand exams alone.

Airborne light detection and ranging (LIDAR) system uses in many field such as military, agriculture, geology and transportation. LIDAR data acquisition consumes several gigabytes from the storage capacity of the system. So, it's necessary to reduce the on board storage capacity as well as its transmission rate. Compression of LIDAR data means reducing the amount of data required to represent the digital data acquired by LIDAR system. This is done by removing data redundancies to achieve compression. In this paper we explain LIDAR system theory and component, compare between lossy and lossless compression techniques then we discuss LIDAR data compression techniques in order to choose a suitable one for LIDAR system that gives high compression ratio with suitable processing time. In most LIDAR data compression techniques classification of LIDAR data is an important issue as a preprocessing step to simplify the data. Using more than one compression algorithms increase the compression ratio but increase the processing time as well.

The joint use of information from hyperspectral image (HSI) data and light detection and ranging (LiDAR) data enables the effective identification and classification of complex land cover categories. Recently, many deep learning classification methods based on the combination of convolutional neural networks (CNN) and Transformer have been used in HSI and LiDAR classification task. However, existing algorithms pay little attention to the fusion of centre pixel features in multimodal data. In this article, we propose a novel centre pixel features enhancement network (CPFENet) for effective cross-modal information fusion. Specifically, this article introduces a centre pixel enhancement module (CPEM), which integrates absolute positional encoding and centre relative position encoding to enhance the expression of centre pixel features. Subsequently, the enhanced features are fed into a novel centre pixel feature cross-attention (CPFCA) module for fusion, which enables the network to focus on the centre pixel to learn spatial and spectral features. Experimental results on three public datasets demonstrate that the performance of the proposed network is better than that of previous state-of-the-art methods.

Good forest management requires comprehensive and reliable inventory data spanning large areas. Forest management has increasingly relied on remote sensing, specifically light detection and ranging (LiDAR). However, due to the high costs associated with data collection and processing, wall-to-wall LiDAR data is rarely obtained for forests. In contrast, multispectral imagery from optical sensors often covers large extents but they fail to capture detail below the forest canopy and do not directly measure structural attributes. To take advantage of the complementary benefits of different sensors, active LiDAR and passive optical sensors have been combined and applied to problem-solving in a forestry context for over a decade. A review of the literature shows that fusion of different sensors has resulted in superior performance relative to individual sensors for classifying and delineating forest areas (up to 20 % accuracy improvement), identifying species (up to 21 % accuracy improvement), and estimating forest volume and biomass (up to 55 % accuracy improvement). In contrast, sensor fusion achieved only minor improvements for tree or forest height estimation (1–7 % accuracy improvement); this is likely because LiDAR alone is already so effective. This review was unable to draw conclusions on the performance of sensor fusion for forest age and productivity assessment due to the limited number of studies. The lack of results in these areas presents an opportunity for future research. The literature clearly demonstrates the utility of integrating LiDAR and optical data for many aspects of forest description. Perhaps the greatest challenge moving forward will be to operationalise the research such that forestry companies and governments can take advantage of the benefits of data fusion.

Hyperspectral imagery (HSI) and light detection and ranging (LiDAR) remote sensing technologies are important ways to obtain surface information. Combining the characteristics and advantages of hyperspectral images and LiDAR DSM data, we propose a classification method for co-classification of hyperspectral images and LiDAR data in this paper. The model uses a dual-branch HSI image and LiDAR data classification method. First, the features of shallow convolution and deep convolution are merged in the spatial feature extraction process of HSI, which combines to focus on more global information. Then the space and spectrum are combined, considering the characteristics of different layers, the network has the characteristics of multiscale. Finally, the LiDAR branch introduces dilated convolution to obtain a larger receptive field and reduce information loss. The classification experimental results on Houston and Trento dataset show the superiority of our proposed method in the collaborative classification of hyperspectral images and LiDAR data.

Hyperspectral image classification has been an active topic of research. In recent years, it has been found that light detection and ranging (LiDAR) data provide a source of complementary information that can greatly assist in the classification of hyperspectral data, in particular when it is difficult to separate complex classes. This is because, in addition to the spatial and the spectral information provided by hyperspectral data, LiDAR can provide very valuable information about the height of the surveyed area that can help with the discrimination of classes and their separability. In the past, several efforts have been investigated for fusion of hyperspectral and LiDAR data, with some efforts driven by the morphological information that can be derived from both data sources. However, a main challenge for the learning approaches is how to exploit the information coming from multiple features. Specifically, it has been found that simple concatenation or stacking of features such as morphological attribute profiles (APs) may contain redundant information. In addition, a significant increase in the number of features may lead to very high-dimensional input features. This is in contrast with the limited number of training samples often available in remote-sensing applications, which may lead to the Hughes effect. In this work, we develop a new efficient strategy for fusion and classification of hyperspectral and LiDAR data. Our approach has been designed to integrate multiple types of features extracted from these data. An important characteristic of the presented approach is that it does not require any regularization parameters, so that different types of features can be efficiently exploited and integrated in a collaborative and flexible way. Our experimental results, conducted using a hyperspectral image and a LiDAR-derived digital surface model (DSM) collected over the University of Houston campus and the neighboring urban area, indicate that the proposed framework for multiple feature learning provides state-of-the-art classification results.

Beijing’s One Million Mu Plain Afforestation Project involves planting large areas with the exotic North American tree species Fraxinus pennsylvanica Marsh (ash). As an exotic tree species, ash is very vulnerable to infestations by the emerald ash borer (EAB), a native Chinese wood borer pest. In the early stage of an EAB infestation, attacked trees show no obvious sign. Once the stand has reached the late damage stage, death occurs rapidly. Therefore, there is a need for efficient early detection methods of EAB stress over large areas. The combination of unmanned aerial vehicle (UAV)-based hyperspectral imaging (HI) with light detection and ranging (LiDAR) is a promising practical approach for monitoring insect disturbance. In this study, we identified the most useful narrow-band spectral HI data and 3D LiDAR data for the early detection of EAB stress in ash. UAV-HI data of different infested stages (healthy, light, moderate and severe) of EAB in the 400–1000 nm range were collected from ash canopies and were processed by Partial Least Squares–Variable Importance in Projection (PLS-VIP) to identify the maximally sensitive bands. Band R678 nm had the highest PLS-VIP scores and the most robust classification ability. We combined this band with band R776 nm to develop an innovative normalized difference vegetation index (NDVI(776,678)) to estimate EAB stress. LiDAR data were used to segment individual trees and supplement the HI data. The new NDVI(776,678) identified different stages of EAB stress, with a producer’s accuracy of 90% for healthy trees, 76.25% for light infestation, 58.33% for moderate infestation, and 100% for severe infestation, with an overall accuracy of 82.90% when combined with UAV-HI and LiDAR.