StrucADT: Generating Structure-Controlled 3D Point Clouds With Adjacency Diffusion Transformer.

The direct extension of 2D image learning to three-dimensional space is 3D point cloud learning. Recently, point cloud learning has shown significant results in 3D shape estimation and semantic segmentation. Despite these advancements, fundamental problems in point cloud learning still pose significant challenges. These problems include representation learning, shape generation, shape segmentation, and shape matching. In this paper, we propose a cognitive self-attention based learning approach to aggregate global representation of 3D shapes from point cloud data. We also integrate 3D point data with a binary tree structure to build a point cloud generator. We further design a novel Generative Adversarial Network (GAN) architecture to generate point clouds resembling the ground truth that could be used for unsupervised learning of 3D shapes. Relying on a self-attention mechanism, our framework, called SAPCGAN, aggregates the global graph features to correct the structural information of 3D shapes in the latent space. Finally, we compare the performance of two gradient penalty methods used in stabilizing the training of our GAN system. We show that our framework has high training efficiency and can provide state-of-the-art results in 3D point cloud generation. The performance of our is demonstrated with both quantitative and qualitative experimental evaluations. Furthermore, the generated 3D point clouds can be segmented into their natural semantic parts, such as, for example the four legs of a chair, the wings of an air plane, or the four wheels of a car.

Point clouds are fundamental in the representation of 3D objects. However, they can also be highly unstructured and irregular. This makes it difficult to directly extend 2D generative models to three-dimensional space. In this article, we cast the problem of point cloud generation as a topological representation learning problem. In order to capture the representative features of 3D shapes in the latent space, we propose a hierarchical mixture model that integrates self-attention with an inference tree structure for constructing a point cloud generator. Based on this, we design a novel Generative Adversarial Network (GAN) architecture that is capable of generating recognizable point clouds in an unsupervised manner. The proposed adversarial framework (SG-GAN) relies on self-attention mechanism and Graph Convolution Network (GCN) to hierarchically infer the latent topology of 3D shapes. Embedding and transferring the global topology information in a tree framework allows our model to capture and enhance the structural connectivity. Furthermore, the proposed architecture endows our model with partially generating 3D structures. Finally, we propose two gradient penalty methods to stabilize the training of SG-GAN and overcome the possible mode collapse of GAN networks. To demonstrate the performance of our model, we present both quantitative and qualitative evaluations and show that SG-GAN is more efficient in training and it exceeds the state-of-the-art in 3D point cloud generation.

The inclusion of thermal infrared (TIR) data in point clouds derived from unmanned aircraft system (UAS) imagery can benefit a variety of applications in which surface temperature and 3D geometry are both important discriminants of feature type and condition. Low resolution and narrow fields of view (FOV) of current consumer-grade TIR cameras on UAS, combined with the lack of sharpness and texture in many image regions, may cause failure or poor results from structure from motion (SfM) photogrammetric software, which has gained widespread use for generating point clouds from UAS imagery. This paper proposes a photogrammetric approach for generating 3D multispectral point clouds utilizing coacquired TIR-RGB images. A 3D point cloud is first generated from the RGB imagery using standard SfM procedures. Then the TIR attributes are assigned to points, where the image coordinates of the points in TIR images are estimated using transformation parameters obtained from co-registration procedures. To obtain RGB-to-TIR transformation parameters, this study tests 3D and 2D co-registration approaches. The latter produces better results due to the challenge of calibrating the TIR camera as required for the 3D approach. This proposed approach is advantageous for generating TIR point clouds without loss of photogrammetric precision compared with solely TIR-based SfM, as the 3D accuracy, point density, and reliability are greatly enhanced.

This paper addresses the generation and visualization of noisy 3D point clouds. The goal is to extract 3D information from image sequences in real-time and to transform the resulting noisy point clouds into a representation with dense surfaces and an intuitive character. Therefore a bridge is built from image sequence analysis to computer graphics. A point cloud is calculated from a single camera by using state of the art algorithms for egomotion estimation and dense 3D reconstruction. Due to inaccuracies in the measurement process and hence of the resulting depth maps, a degree of uncertainty of the 3D data is defined. This additional information can be used to filter the point cloud in space and time. We present an approach to filter and polygonize weighted point clouds and visualize it by mapping image texture on the resulting surface.

In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized probability density, thereby moving sampled points toward the high-likelihood regions. Our model directly predicts the gradient of the log density field and can be trained with a simple objective adapted from score-based generative models. We show that our method can reach state-of-the-art performance for point cloud auto-encoding and generation, while also allowing for extraction of a high-quality implicit surface. Code is available at https://github.com/RuojinCai/ShapeGF .

Learning geometry-aware joint latent space for simultaneous multimodal shape generation

Point cloud shape correspondence aims at accurately mapping one point cloud to another point cloud with various 3D shapes. Since point clouds are usually sparse, disordered, irregular, and with diverse shapes, it is challenging to learn consistent point cloud representations and achieve the accurate matching of different point cloud shapes. To address the above issues, we propose a Hierarchical Shape-consistent TRansformer for unsupervised point cloud shape correspondence (HSTR), including a multi-receptive-field point representation encoder and a shape-consistent constrained module in a unified architecture. The proposed HSTR enjoys several merits. In the multi-receptive-field point representation encoder, we set progressively larger receptive fields in different blocks to simultaneously consider the local structure and the long-range context. In the shape-consistent constrained module, we design two novel shape selective whitening losses, which can complement each other to achieve suppression of features sensitive to shape change. Extensive experimental results on four standard benchmarks demonstrate the superiority and generalization ability of our approach to existing methods at the similar model scale, and our method achieves the new state-of-the-art results.

Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning

Unmanned aerial vehicles (UAVs) or drones are lightly weighted platforms with numerous advantages and can acquire very high-resolution data rapidly with less expenditure. However, the processing of very high-resolution data acquired through the sensors mounted on the UAV platform is a major challenge, and the commercial UAV data processing software is expensive. These software packages require high-configuration hardware and follow the general workflow of conventional photogrammetric approach for image alignment, point cloud generation, mesh generation, DSM, and ortho-mosaic generation. The integration of computer vision algorithms into digital photogrammetric techniques has made it possible to process the high-resolution datasets quickly. A wide variety of open-source software utilities are available that can perform these tasks individually; however, an integrated solution is always desirable. An attempt is made to develop an open-source tool for photogrammetric processing of UAV data using open-source libraries in Python. The software provides a complete workflow for photogrammetric processing of UAV data ranging from camera calibration, point cloud generation from selective frames based on B/H ratio, direct georeferencing of generated point cloud and digital elevation model (DEM) generation. The developed software is compatible with both 32-bit and 64-bit operating systems and low configuration hardware.

Light Detection And Ranging (LiDAR) is an active laser measuring technology that combines laser scanning and Position and Orientation System (POS) in imaging for generation of accurate and dense 3D point clouds, Digital Elevation Models (DEMs) and Digital Surface Models (DSMs). Other value addition products such as contours, slope maps, tree and building height models and cut-and-fill models can also be produced from the primary LiDAR point cloud data.

Abstract. Beach and fluvial systems are highly dynamic environments, being constantly modified by the action of different natural and anthropic phenomena. To understand their behaviour and to support a sustainable management of these fragile environments, it is very important to have access to cost-effective tools. These methods should be supported on cutting-edge technologies that allow monitoring the dynamics of the natural systems with high periodicity and repeatability at different temporal and spatial scales instead the tedious and expensive field-work that has been carried out up to date. The work herein presented analyses the potential of terrestrial photogrammetry to describe beach morphology. Data processing and generation of high resolution 3D point clouds and derived DEMs is supported by the commercial Agisoft PhotoScan. Model validation is done by comparison of the differences in the elevation among the photogrammetric point cloud and the GPS data along different beach profiles. Results obtained denote the potential that the photogrammetry 3D modelling has to monitor morphological changes and natural events getting differences between 6 and 25 cm. Furthermore, the usefulness of these techniques to control the layout of a fluvial system is tested by the performance of some modeling essays in a hydraulic pilot channel.

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications [1,2].METHODS & RESULTS: While new approaches for deep neural networks are rapidly developing [1], the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ [3], can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments [1], purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo [5], enables to perform ML for change analysis in multitemporal point clouds [6]. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo). REFERENCES:[1] Esmorís Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.[2] Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.112772 [3] HELIOS++: https://github.com/3dgeo-heidelberg/helios[4] VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d[5] py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo[6] Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034

As a mainstream form of 3D data, point clouds are widely used in computer vision for tasks such as segmentation, classification, and target detection due to their simple representation method and high stability and accuracy. Considering the issues of noise points and uneven point distribution in current generation models, we propose a novel adversarial network model, SAC-GAN, which incorporates both a self-attention mechanism and a curvature learning mechanism. Firstly, the feature enhancement module and the pre-processing module, which are based on the ShapeNetCore open dataset, are designed on the generator to enhance the authenticity of local geometric details in the generated point cloud. Secondly, the loss function of the discriminator is adjusted to combine the traditional Wasserstein distance with the normal vector of key points to guide the generation of subtle features of point clouds and improve the quality and consistency of generated point clouds; Finally, to enhance the discriminator's capacity to extract both local and global features, a self-attention mechanism is introduced. This enhances the discriminator's capacity to discern the details of the generated point cloud and offers superior feedback to the generator. Experimental results indicate that the proposed point cloud generation model outperforms existing methods, including TreeGAN, SP-GAN, PDGN, and WarpingGAN, in terms of generation quality. Specifically, the model achieves a reduction in JSD by 4.24, a decrease in MMD by 0.8, and an increase in COV by 1.25%. It can be proven that the point cloud generated by SAC-GAN model has a good performance in shape integrity and authenticity.

Abstract. The research aims to overcome the limitations of physical conservations of architectural heritage using digital interactions and gamification. Physical conservation is limited by practical factors like materials availability or durability of the originally used materials. On the other hand, digital 3D interactive models do not face all these constraints. Therefore, this project demonstrates how an ancient heritage temple, the Thian Hock Keng in Singapore, benefits from this expanded version of digital interactions and gamification. Digital Heritage (DH) methods, such as the generation of 3d point clouds and construction of 3D polygonal models, are used to create a hyper realistic scan of the heritage site. On top of a 3d walkthrough of the heritage space, classical narrative storytelling technique is used to keep the player engaged in the gameplay with a 3-act structured story plot. The outcome not only covers wider perspectives of the history of this space, but it also gains the advantage in its presentation through gaming, where the audience is fully immersed in the virtual temple space as a first-person player. The gamification of the 3D Thian Hock Keng temple serves as a case study to support further research on expansions of digital heritage conservations through interactions.