Multiview 3D sensing and analysis for high quality point cloud reconstruction

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

Estimation of plant parameters is prerequisite for fast and precise management of tomato crops on commercial farm level. Destructive estimation methods may be accurate but not suitable when large farm size, spatio-temporal variability or short period of time to obtain information are considered. Therefore, this study was aimed to estimate leaf area (LA) in mature tomato plants from reconstructed geometric plant model based on 3-dimensional (3D) point cloud. A 2D laser scanner was placed on a programable linear conveyor system to move at controlled forward speed. To achieve a high density 3D point cloud of whole plant, close range (< 2 m) scanning was performed at 10 mm/s forward speed. Plant parameters capturing points per plant (PPP), plant height, plant volume, and LA were extracted from the 3D point clouds. Destructive method was utilized in laboratory for reference LA data collection, whereas other required reference data (plant height and canopy area) were measured in-situ. Linear regression models were developed and cross validated to estimate LA using variables from the 3D point clouds and reference LA. Among the models, plant height showed strong correlation with reference LA (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.90, RMSE = 40 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and followed by PPP (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.89, RMSE = 45 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), voxel grid approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.56, RMSE = 58 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and slicing approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.44, RMSE = 70 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ). Results indicate that mobile laser scanning could be a reliable non-destructive LA estimation tool for tomato and may be adapted by the commercial tomato growers for profitable management decisions.

Abstract. Forest inventory provides comprehensive information about the geometric and biometric state of forests as well as vegetated areas. In this study, a point-based 3D method is presented for tree detection as well as measuring of structural properties of forests such as the tree height, tree position and canopy area using high resolution point cloud which is provided by an Unmanned Aerial Vehicle (UAV)-based LiDAR sensor. The proposed method is based on the density of point cloud and 2D and 3D distance measurements. It includes three main steps as pre-processing, tree detection, and extraction of tree structural attributes. After generating a canopy height model, an image is created based on the density of point cloud. Next, points are classified based on 2D and 3D distance measurements, sequentially, from the highest to the lowest. According to the results, the rate of tree detection is about 95% and the main structural parameters of a tree such as the position, height, area and length of the canopy are estimated with the accuracy of 1.97 m, 0.36 m, 12.78 m2 and 0.79 m, respectively.

Real-time embedded vision tasks require extraction of complex geometric and morphological features from the raw 3D point cloud acquired using range scanning systems like lidar, radar etc. and depth cameras. Such applications are found in autonomous navigation, surveying, 3D mapping and localization tasks such as automatic target recognition (ATR). Typically, a dataset acquired during surveying by remote sensing lidar scanners, known as point cloud, is (1) huge in size and requires a big chunk of memory for processing at a single instance and, (2) experiences missing information due to rapid change in orientation of the sensor while scanning. In our work, we are addressing both the issues combinedly by proposing an online point cloud super-resolution approach for translating a low dimensional point cloud to a high dimensional dense point cloud by learning dictionaries in the low-dimensional subspace. We are presenting our approach for an urban road scenario by reconstructing dense point clouds of 3D objects and comparing results based on PSNR and Hausdorff distance.

The 3D modeling of urban buildings has become a common research area in various disciplines such as photogrammetry and computer vision, with different applications such as intelligent city management, navigation of self-driving cars and architecture, just to name a few. The objective of this study is to produce a 3D model of the external facade of the buildings with the required precision, accuracy and level of detail according to the user’s requirements, while minimizing time and cost. This research focuses on the production of 3D models for blocks of residential buildings in Tehran, Iran. The Insta 360 One X2 spherical camera is selected to capture the data due to its low cost and 360 × 180° field of view. The camera specifications have facilitated more efficient data collection in terms of both time and cost. The proposed modeling method is based on extracting lines of external walls through the utilization of the point cloud density concept. Initially, photogrammetric point clouds are generated in with a reconstruction precision of 0.24 m from spherical camera images. In the next step, the 3D point cloud is projected into a 2D point cloud by setting the height component to zero. The 2D point cloud is then rotated based on the direction angle determined by the Hough transform so that the perpendicular walls are parallel to the axes of the coordinate system. Next, a 2D point cloud density analysis is performed by voxelizing the point cloud and counting the number of points in each voxel in both the horizontal and vertical directions. By determining the peaks in the density plot, the lines of the external vertical and horizontal walls are extracted. To extract the diagonal external walls, the density analysis is performed in the direction of the first principal component. Finally, by determining the height of each wall in the point cloud, a 3D model is created at the level of detail one. The resulting model has a precision of 0.32 m compared to real sizes, and the 2D plan has a precision of 0.31 m compared to the ground truth map. The use of the spherical camera and point cloud density analysis makes this method efficient and cost-effective, making it a promising approach for future urban modeling projects.

In this paper we explore the robustness of histogram features extracted from 3D point clouds of human subjects for gender classification. Experiments are conducted using point clouds drawn from the Civilian American and European Surface Anthropometry Resource Project (CAESAR anthropometric database provided by the Air Force Research Laboratory (AFRL) Human Effectiveness Directorate and SAE International). This database contains approximately 4400 high resolution LIDAR whole body scans of carefully posed human subjects. Features are extracted from each point cloud by embedding the cloud in series of cylindrical shapes and computing a point count for each cylinder that characterizes a region of the subject. These measurements define rotationally invariant histogram features that are processed by a classifier to label the gender of each subject. Preliminary results using cylinder sizes defined by human experts demonstrate that gender can be predicted with 98% accuracy for the type of high density point cloud found in the CAESAR database. In our previous has shown that when point cloud densities are reduced to levels that might be obtained using stand-off sensors; gender classification accuracy degrades. In this paper we show the results of how the classification accuracy degrades as a function of center of mass displacements.

Reconstruction 3D objects from a single view is an important research direction in computer vision. With the development of deep learning technology, a single-view 3D reconstruction based on deep learning has made remarkable progress in recent years. In reviewing 3D object reconstruction based on deep learning, firstly, the research progress of 3D reconstruction methods based on deep learning is analyzed in detail according to different representations of 3D objects, i.e., point cloud, voxel, and mesh. Secondly, the standard 3D reconstruction datasets and evaluation criteria are summarized. Finally, the challenges of 3D object reconstruction based on deep learning are summarized.

The PFuji-Size dataset is comprised of a collection of 3D point clouds of Fuji apple trees (Malus domestica Borkh. cv. Fuji) scanned at different maturity stages and annotated for fruit detection and size estimation. Structure-from-motion and multi-view stereo techniques were used to generate the 3D point clouds of 6 complete Fuji apple trees containing a total of 615 apples. The resulting point clouds were 3D segmented by identifying the 3D points corresponding to each apple (3D instance segmentation), obtaining a single point cloud for each apple. All segmented apples were labelled with ground truth diameter annotations. Since the data was acquired in field conditions and at different maturity stages, the set includes different fruit diameters -from 26.9 mm to 94.8 mm- and different fruit occlusion percentages due to foliage. In addition, 25 apples were photographed 360° in laboratory conditions, obtaining high resolution 3D point clouds of this sub-set. To the best of the authors’ knowledge, this is the first publicly available dataset for apple size estimation in field conditions. This dataset was used to evaluate different fruit size estimation methods in the research article titled “In-field apple size estimation using photogrammetry-derived 3D point clouds: comparison of 4 different methods considering fruit occlusion” (Gené-Mola et al., 2021).

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

In this paper we introduce a Lidar based real time and accurate self localization approach for self driving vehicles (SDV) in high resolution 3D point cloud maps of the environment obtained through Mobile Laser Scanning (MLS). Our solution is able to robustly register the sparse point clouds of the SDVs to the dense MLS point cloud data, starting from a GPS based initial position estimation of the vehicle. The main steps of the method are robust object extraction and transformation estimation based on multiple keypoints extracted from the objects, and additional semantic information derived from the MLS based map. We tested our approach on roads with heavy traffic in the downtown of a large city with large GPS positioning errors, and showed that the proposed method enhances the matching accuracy with an order of magnitude. Comparative tests are provided with various keypoint selection strategies, and against a state-of-the-art technique.

&amp;lt;p&amp;gt;The use of high resolution 3D point clouds and digital terrain models (DTM) from laserscanning or photogrammetry becomes more and more state of the art in landslide studies. Based on a multi-temporal terrestrial laserscanning (TLS) dataset of the deep-seated compound rockslide Laatsch, South Tyrol, we present a multi-method approach to characterize processes such as sliding, falling, toppling, and flows. Sliding is the predominant process of the Laatsch rockslide, accompanied by secondary processes such as rockfall, debris flows and erosion. The presented methods are applicable to all kind of 3D point clouds and not limited to TLS data. For remote sensing-based landslide analyses a distinction between two classes of surface processes is necessary: i) processes where the original surface is destroyed and no correlations between the shape and texture of the pre- and post-failure surfaces can be found (falls, rapid flows, rapid slides) and ii) processes where the surface is displaced without major surface changes (slow slides, slow flows and toppling). For processes where the original surface is destroyed, the distance between the pre- and post-failure terrain surface is measured with the aim to delineate the scarp and depositional area, and to quantify the failure volume as well as the scarp thickness. With DTMs of differences (DoD), the distance is measured along the plumb line. DoDs can be used to quickly and reliably assess the volume and extent of fall processes on flat to moderate slopes. For steep or even overhanging terrain, a 3D distance measurement approach must be used, where the distance is measured along the local surface normal. After 3D distance measurement, the volume of steep scarp areas can be calculated by first rotating, the point cloud into the horizontal plane (by making use of the average surface normal) and by interpolating the rotated 3D distance measurement values into a grid. Summing up the distances and multiplying with the cell area of the grid yields the scrap rupture volume. Remote sensing-based analyses of sliding and toppling processes are more complex compared to fall processes because the displaced surface patch must be detected in both surveys. Displacement analyses based on image correlation of ambient occlusion shaded relief images, together with DTMs of both epochs, are used to analyse the displacement of the entire rockslide area. The result is a map with displacement vectors. Disadvantages of image correlation are the coarse spatial resolution and the inability, as it is a 2.5D approach, to deal with steep slope parts. To analyse the displacement and toppling of steep rock walls a combination of the 3D distance measurement approach and an iterative closest point (ICP) based approach is applied. The 3D distance measurement values are clustered and used for a segmentation of the point cloud. In a next step, the ICP is applied on each of the resulting segments. This approach can deal with 3D displacements. The results are still sensitive towards the geometric contrast within the segments and not fully automated yet.&amp;lt;/p&amp;gt;

Towards semi-automatic discontinuity characterization in rock tunnel faces using 3D point clouds

The advancement of deep learning technology has been concentrating on deploying end-to-end solutions using high dimensional data, such as images. Recently, a number of methods have been proposed for reconstructing 3D objects using deep learning. One such technique is the method that involves recovering 3D objects as voxel grids using one or multiple images. However, there has been very little work directed towards the generation of 3D objects represented by a set of points, i.e. point cloud, from voxel grids which are ambiguous and coarse. The development of a deep learning model that generates point clouds with details from coarse voxel grids has numerous benefits to quality of life. For example, design professionals can use this model to generate detailed 3D point clouds using a sketched and coarse voxel to enable their creativities. This paper presents an algorithm to generate point clouds from voxel grids. The algorithm explicitly loads 3D objects into voxels without projection operations and associated information loss. To obtain a comprehensive understanding of the voxel grid, the grid is analyzed through various angles, as inspired by how humans observe 3D objects. The features from various angles are passed into a GRU layer to extract patterns across views, which will then be passed to a channel-wise convolutional layer and graph convolution to generated the predicted point cloud. The experimental result of the algorithm indicates that the algorithm is capable of generating high-quality point clouds by understanding the semantic features of the voxel grid.

In recent years, there has been significant growth in the ubiquity and popularity of three-dimensional (3D) point clouds, with an increasing focus on the classification of 3D point clouds. To extract richer features from point clouds, many researchers have turned their attention to various point set regions and channels within irregular point clouds. However, this approach has limited capability in attending to crucial regions of interest in 3D point clouds and may overlook valuable information from neighboring features during feature aggregation. Therefore, this paper proposes a novel 3D point cloud classification method based on global attention and adaptive graph convolution (Att-AdaptNet). The method consists of two main branches: the first branch computes attention masks for each point, while the second branch employs adaptive graph convolution to extract global features from the point set. It dynamically learns features based on point interactions, generating adaptive kernels to effectively and precisely capture diverse relationships among points from different semantic parts. Experimental results demonstrate that the proposed model achieves 93.8% in overall accuracy and 90.8% in average accuracy on the ModeNet40 dataset.