Joint Classification of Hyperspectral and LiDAR Data Based on Mamba

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data is widely used for land cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image differences, and the dimensions and feature distributions of HSI and LiDAR are highly dissimilar. This makes it challenging to represent and correlate semantic information from multimodal data. Current methods for classifying pixel-by-pixel features, which rely on cascaded or attention-based fusion, cannot effectively use multimodal features. To achieve accurate classification results, extracting and fusing similar high-order semantic information and complementary discriminative information contained in multimodal data is vital. In this paper, we propose a cross-modal semantic enhancement network (CMSE) for multimodal semantic information mining and fusion. Our proposed CMSE framework extracts features from the image on multiple scales, capturing more representative local sparse features with different sizes of convolution kernels. To represent high-level semantic features related to land cover, we establish a Gaussian-weighted matrix and semantically transform the spatial and spectral features of distinct branches. Finally, we build a multi-level residual fusion module to incrementally fuse spectral features from HSI and elevation features from LiDAR. Additionally, we introduce a cross-modal semantically constrained loss to guide multimodal semantic feature alignment. We evaluate our approach on three multimodal remote sensing datasets, namely the Houston2013, Trento, and MUUFL datasets. The experimental results demonstrate that our proposed CMSE model achieves superior performance in terms of accuracy and robustness compared to other related deep networks.

Hyperspectral image classification has been an active topic of research. In recent years, it has been found that light detection and ranging (LiDAR) data provide a source of complementary information that can greatly assist in the classification of hyperspectral data, in particular when it is difficult to separate complex classes. This is because, in addition to the spatial and the spectral information provided by hyperspectral data, LiDAR can provide very valuable information about the height of the surveyed area that can help with the discrimination of classes and their separability. In the past, several efforts have been investigated for fusion of hyperspectral and LiDAR data, with some efforts driven by the morphological information that can be derived from both data sources. However, a main challenge for the learning approaches is how to exploit the information coming from multiple features. Specifically, it has been found that simple concatenation or stacking of features such as morphological attribute profiles (APs) may contain redundant information. In addition, a significant increase in the number of features may lead to very high-dimensional input features. This is in contrast with the limited number of training samples often available in remote-sensing applications, which may lead to the Hughes effect. In this work, we develop a new efficient strategy for fusion and classification of hyperspectral and LiDAR data. Our approach has been designed to integrate multiple types of features extracted from these data. An important characteristic of the presented approach is that it does not require any regularization parameters, so that different types of features can be efficiently exploited and integrated in a collaborative and flexible way. Our experimental results, conducted using a hyperspectral image and a LiDAR-derived digital surface model (DSM) collected over the University of Houston campus and the neighboring urban area, indicate that the proposed framework for multiple feature learning provides state-of-the-art classification results.

Hyperspectral imagery (HSI) and light detection and ranging (LiDAR) remote sensing technologies are important ways to obtain surface information. Combining the characteristics and advantages of hyperspectral images and LiDAR DSM data, we propose a classification method for co-classification of hyperspectral images and LiDAR data in this paper. The model uses a dual-branch HSI image and LiDAR data classification method. First, the features of shallow convolution and deep convolution are merged in the spatial feature extraction process of HSI, which combines to focus on more global information. Then the space and spectrum are combined, considering the characteristics of different layers, the network has the characteristics of multiscale. Finally, the LiDAR branch introduces dilated convolution to obtain a larger receptive field and reduce information loss. The classification experimental results on Houston and Trento dataset show the superiority of our proposed method in the collaborative classification of hyperspectral images and LiDAR data.

The joint classification of hyperspectral imaging (HSI) and Light Detection and Ranging (LiDAR) data holds significant importance for various practical uses, including urban mapping, mineral prospecting, and ecological observation. Achieving robust and transferable feature representations is essential to fully leverage the complementary properties of HSI and LiDAR modalities. However, existing methods are often constrained to scene-specific training and lack generalizability across datasets, limiting their discriminative power. To tackle this challenge, we introduce a new dual-phase approach for the combined classification of HSI and LiDAR data. Initially, a transformer-driven network is trained on various HSI-only datasets to extract universal spatial–spectral features. In the second stage, LiDAR data is incorporated as a task-specific prompt to adapt the model to HSI-LiDAR scenes and enable effective multimodal fusion. Through extensive testing on three benchmark datasets, our framework proves highly effective, outperforming all competing approaches.

The joint use of information from hyperspectral image (HSI) data and light detection and ranging (LiDAR) data enables the effective identification and classification of complex land cover categories. Recently, many deep learning classification methods based on the combination of convolutional neural networks (CNN) and Transformer have been used in HSI and LiDAR classification task. However, existing algorithms pay little attention to the fusion of centre pixel features in multimodal data. In this article, we propose a novel centre pixel features enhancement network (CPFENet) for effective cross-modal information fusion. Specifically, this article introduces a centre pixel enhancement module (CPEM), which integrates absolute positional encoding and centre relative position encoding to enhance the expression of centre pixel features. Subsequently, the enhanced features are fed into a novel centre pixel feature cross-attention (CPFCA) module for fusion, which enables the network to focus on the centre pixel to learn spatial and spectral features. Experimental results on three public datasets demonstrate that the performance of the proposed network is better than that of previous state-of-the-art methods.

Foresters are increasingly interested in remote sensing data because they provide an overview of landscape conditions, which is impractical with field sample data alone. Light Detection and Ranging (LiDAR) provides exceptional spatial detail of forest structure, but difficulties in processing LiDAR data have limited their application beyond the research community. Another obstacle to operational use of LiDAR data has been the high cost of data collection. Our objectives in this study were to summarize, at the stand level, both LiDAR- and Landsat (satellite)-based predictions of some common structural and volume attributes and to compare the cost of obtaining such summaries with those obtained through traditional stand exams. We found that the accuracy and cost of a LiDAR-based inventory summarized at the stand level was comparable to traditional stand exams for structural attributes. However, the LiDAR data were able to provide information across a much larger area than the stand exams alone.

In this study, we develop a novel deep hierarchical vision transformer (DHViT) architecture for hyperspectral and light detection and ranging (LiDAR) data joint classification. Current classification methods have limitations in heterogeneous feature representation and information fusion of multi-modality remote sensing data (e.g., hyperspectral and LiDAR data), these shortcomings restrict the collaborative classification accuracy of remote sensing data. The proposed deep hierarchical vision transformer architecture utilizes both the powerful modeling capability of long-range dependencies and strong generalization ability across different domains of the transformer network, which is based exclusively on the self-attention mechanism. Specifically, the spectral sequence transformer is exploited to handle the long-range dependencies along the spectral dimension from hyperspectral images, because all diagnostic spectral bands contribute to the land cover classification. Thereafter, we utilize the spatial hierarchical transformer structure to extract hierarchical spatial features from hyperspectral and LiDAR data, which are also crucial for classification. Furthermore, the cross attention (CA) feature fusion pattern could adaptively and dynamically fuse heterogeneous features from multi-modality data, and this contextual aware fusion mode further improves the collaborative classification performance. Comparative experiments and ablation studies are conducted on three benchmark hyperspectral and LiDAR datasets, and the DHViT model could yield an average overall classification accuracy of 99.58%, 99.55%, and 96.40% on three datasets, respectively, which sufficiently certify the effectiveness and superior performance of the proposed method.

Civil engineering includes a wide range of applications that require three-dimensional (3D) information. With the advent of LiDAR (Light Detection And Ranging) technology, obtaining 3D terrain information accurately in a cost and time effective manner became more popular. In this paper, three case studies demonstrate the roles and applications of airborne LiDAR data including highway, transmission line and hydropower dam development projects. In first case study, the highway alignment was considerably changed after the initial design stage. LiDAR data was able to do the re-alignment survey quickly, allowing for prompt changes in design. In the next case study, a comparison is made between conventional transmission route alignment design and the design using PLS-CADD with LiDAR data. Utilizing the LiDAR data in PLSCADD allows transmission line engineers to model the construction costs of several alignments options effectively and efficiently compared to the conventional method. The last case study uses LiDAR for obtaining the necessary information for proposed dam. Using LiDAR data maximizes topological accuracy and volume measurements while minimizing survey time and costs. Furthermore, utilizing the multi-temporal LiDAR data can reveal valuable information such as topography changes. LiDAR survey technology has proven that it is the best method to capture accurate data from the earth surface which provides maximum density of topographical & digital imagery information for various applications.

Light detection and ranging (LiDAR) data classification provides useful thematic maps for numerous geospatial applications. Several methods and algorithms have been proposed recently for LiDAR data classification. Most studies focused on object-based analysis because of its advantages over per-pixel-based methods. However, several issues, such as parameter optimization, attribute selection, and development of transferable rulesets, remain challenging in this topic. This study contributes to LiDAR data classification by developing an approach that integrates ant colony optimization (ACO) and rule-based classification. First, LiDAR-derived digital elevation and digital surface models were integrated with high-resolution orthophotos. Second, the processed raster was segmented with the multiresolution segmentation method. Subsequently, the parameters were optimized with a supervised technique based on fuzzy analysis. A total of 20 attributes were selected based on general knowledge on the study area and LiDAR data; the best subset containing 12 attributes was then selected via ACO. These attributes were utilized to develop rulesets through the use of a decision tree algorithm, and a thematic map was generated for the study area. Results revealed the robustness of the proposed method, which has an overall accuracy of ~95% and a kappa coefficient of 0.94. The rule-based approach with all attributes and the k nearest neighbor (KNN) classification method were applied to validate the results of the proposed method. The overall accuracy of the rule-based method with all attributes was ~88% (kappa = 0.82), whereas the KNN method had an overall accuracy of <;70% and produced a poor thematic map. The selection of the ACO algorithm was justified through a comparison with three well-known feature selection methods. On the other hand, the transferability of the developed rules was evaluated by using a second LiDAR dataset at another study area. The overall accuracy and the kappa index for the second study area were 92% and 0.90, respectively. Overall, the findings indicate that the selection of a subset with significant attributes is important for accurate LiDAR data classification with object-based methods.

Accurate estimation of the fraction of absorbed photosynthetically active radiation (fPAR) is important for maize growth and yield estimations. Light detection and ranging (LiDAR)-derived canopy vertical structural and hyperspectral image-derived vegetation spectral information are complementary for vegetation fPAR estimation. This study explores the potential of artificial neural networks (ANNs) with two types of data to estimate maize fPAR. First, 45 metrics were derived from LiDAR data and 13 from a hyperspectral image. Then, the ANNs and stepwise multiple linear regression (SMLR) methods were used to estimate the fPAR. Finally, model validity was assessed using in-situ data. Results showed that the ANNs performed better in fPAR inversion (R 2 = 0.910, adj. R 2 = 0.921, RMSE = 0.046, RRMSE = 0.056, where R 2 is the coefficient of determination, adj. R 2 the adjusted coefficient of determination, RMSE the root mean squared error, and RRMSE the relative root mean squared error) than SMLR (R 2 = 0.638, adj. R 2 = 0.609, RMSE = 0.077, RRMSE = 0.092) and SMLR with the natural logarithm of data (R 2 = 0.855, adj. R 2 = 0.825, RMSE = 0.067, RRMSE = 0.081). This study is helpful for guiding the accurate estimation of maize fPAR using remote sensing techniques.

Beijing’s One Million Mu Plain Afforestation Project involves planting large areas with the exotic North American tree species Fraxinus pennsylvanica Marsh (ash). As an exotic tree species, ash is very vulnerable to infestations by the emerald ash borer (EAB), a native Chinese wood borer pest. In the early stage of an EAB infestation, attacked trees show no obvious sign. Once the stand has reached the late damage stage, death occurs rapidly. Therefore, there is a need for efficient early detection methods of EAB stress over large areas. The combination of unmanned aerial vehicle (UAV)-based hyperspectral imaging (HI) with light detection and ranging (LiDAR) is a promising practical approach for monitoring insect disturbance. In this study, we identified the most useful narrow-band spectral HI data and 3D LiDAR data for the early detection of EAB stress in ash. UAV-HI data of different infested stages (healthy, light, moderate and severe) of EAB in the 400–1000 nm range were collected from ash canopies and were processed by Partial Least Squares–Variable Importance in Projection (PLS-VIP) to identify the maximally sensitive bands. Band R678 nm had the highest PLS-VIP scores and the most robust classification ability. We combined this band with band R776 nm to develop an innovative normalized difference vegetation index (NDVI(776,678)) to estimate EAB stress. LiDAR data were used to segment individual trees and supplement the HI data. The new NDVI(776,678) identified different stages of EAB stress, with a producer’s accuracy of 90% for healthy trees, 76.25% for light infestation, 58.33% for moderate infestation, and 100% for severe infestation, with an overall accuracy of 82.90% when combined with UAV-HI and LiDAR.

Currently, the worsening impacts of urbanizations have been impelled to the importance of monitoring and management of existing urban trees, securing sustainable use of the available green spaces. Urban tree species identification and evaluation of their roles in atmospheric Carbon Stock (CS) are still among the prime concerns for city planners regarding initiating a convenient and easily adaptive urban green planning and management system. A detailed methodology on the urban tree carbon stock calibration and mapping was conducted in the urban area of Brussels, Belgium. A comparative analysis of the mapping outcomes was assessed to define the convenience and efficiency of two different remote sensing data sources, Light Detection and Ranging (LiDAR) and WorldView-3 (WV-3), in a unique urban area. The mapping results were validated against field estimated carbon stocks. At the initial stage, dominant tree species were identified and classified using the high-resolution WorldView3 image, leading to the final carbon stock mapping based on the dominant species. An object-based image analysis approach was employed to attain an overall accuracy (OA) of 71% during the classification of the dominant species. The field estimations of carbon stock for each plot were done utilizing an allometric model based on the field tree dendrometric data. Later based on the correlation among the field data and the variables (i.e., Normalized Difference Vegetation Index, NDVI and Crown Height Model, CHM) extracted from the available remote sensing data, the carbon stock mapping and validation had been done in a GIS environment. The calibrated NDVI and CHM had been used to compute possible carbon stock in either case of the WV-3 image and LiDAR data, respectively. A comparative discussion has been introduced to bring out the issues, especially for the developing countries, where WV-3 data could be a better solution over the hardly available LiDAR data. This study could assist city planners in understanding and deciding the applicability of remote sensing data sources based on their availability and the level of expediency, ensuring a sustainable urban green management system.

High-value timber species play an important economic role in forest management. The individual tree information for such species is necessary for practical forest management and for conservation purposes. Digital aerial photogrammetry derived from an unmanned aerial vehicle (UAV-DAP) can provide fine spatial and spectral information, as well as information on the three-dimensional (3D) structure of a forest canopy. Light detection and ranging (LiDAR) data enable area-wide 3D tree mapping and provide accurate forest floor terrain information. In this study, we evaluated the potential use of UAV-DAP and LiDAR data for the estimation of individual tree location and diameter at breast height (DBH) values of large-size high-value timber species in northern Japanese mixed-wood forests. We performed multiresolution segmentation of UAV-DAP orthophotographs to derive individual tree crown. We used object-based image analysis and random forest algorithm to classify the forest canopy into five categories: three high-value timber species, other broadleaf species, and conifer species. The UAV-DAP technique produced overall accuracy values of 73% and 63% for classification of the forest canopy in two forest management sub-compartments. In addition, we estimated individual tree DBH Values of high-value timber species through field survey, LiDAR, and UAV-DAP data. The results indicated that UAV-DAP can predict individual tree DBH Values, with comparable accuracy to DBH prediction using field and LiDAR data. The results of this study are useful for forest managers when searching for high-value timber trees and estimating tree size in large mixed-wood forests and can be applied in single-tree management systems for high-value timber species.

We present a remote sensing pipeline that processes LiDAR (Light Detection And Ranging) data through machine & deep learning for the application of archeological feature detection on big geo-spatial data platforms such as e.g. IBM PAIRS Geoscope. Today, archeologists get overwhelmed by the task of visually surveying huge amounts of (raw) LiDAR data in order to identify areas of interest for inspection on the ground. We showcase a software system pipeline that results in significant savings in terms of expert productivity while missing only a small fraction of the artifacts. Our work employs artificial neural networks in conjunction with an efficient spatial segmentation procedure based on domain knowledge. Data processing is constraint by a limited amount of training labels and noisy LiDAR signals due to vegetation cover and decay of ancient structures. We aim at identifying geo-spatial areas with archeological artifacts in a supervised fashion allowing the domain expert to flexibly tune parameters based on her needs.