Segmentation and Recognition of Highway Assets Using Image-Based 3D Point Clouds and Semantic Texton Forests

The hyperloop transportation system has emerged as an innovative next-generation transportation system. In this system, a capsule-type vehicle inside a sealed near-vacuum tube moves at 1000 km/h or more. Not only must this transport tube span over long distances, but it must be clear of potential hazards to vehicles traveling at high speeds inside the tube. Therefore, an automated infrastructure anomaly detection system is essential. This study sought to confirm the applicability of advanced sensing technology such as Light Detection and Ranging (LiDAR) in the automatic anomaly detection of next-generation transportation infrastructure such as hyperloops. To this end, a prototype two-dimensional LiDAR sensor was constructed and used to generate three-dimensional (3D) point cloud models of a tube facility. A technique for detecting abnormal conditions or obstacles in the facility was used, which involved comparing the models and determining the changes. The design and development process of the 3D safety monitoring system using 3D point cloud models and the analytical results of experimental data using this system are presented. The tests on the developed system demonstrated that anomalies such as a 25 mm change in position were accurately detected. Thus, we confirm the applicability of the developed system in next-generation transportation infrastructure.

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

Reconstructing three-dimensional (3D) point cloud model of maize plants can provide reliable data for its growth observation and agricultural machinery research. The existing data collection systems and registration methods have low collection efficiency and poor registration accuracy. A point cloud registration method for maize plants based on conical surface fitting—iterative closest point (ICP) with automatic point cloud collection platform was proposed in this paper. Firstly, a Kinect V2 was selected to cooperate with an automatic point cloud collection platform to collect multi-angle point clouds. Then, the conical surface fitting algorithm was employed to fit the point clouds of the flowerpot wall to acquire the fitted rotation axis for coarse registration. Finally, the interval ICP registration algorithm was used for precise registration, and the Delaunay triangle meshing algorithm was chosen to triangulate the point clouds of maize plants. The maize plant at the flowering and kernel stage was selected for reconstruction experiments, the results show that: the full-angle registration takes 57.32 s, and the registration mean distance error is 1.98 mm. The measured value’s relative errors between the reconstructed model and the material object of maize plant are controlled within 5%, the reconstructed model can replace maize plants for research.

IoT-based low-cost 3D mapping using 2D Lidar for different materials

Stereo cameras, laser rangers and other time-of-flight ranging devices are utilized with increasing frequency as they can provide information in the 3D plane. The ability to perform real-time registration of the 3D point clouds obtained from these sensors is important in many applications. However, the tasks of locating accurate and dependable correspondences between point clouds and registration can be quite slow. Furthermore, any algorithm must be robust against artifacts in 3D range data as sensor motion, reflection and refraction are commonplace. The SIFT feature detector is a robust algorithm used to locate features, but cannot be extended directly to the 3D range point clouds since it requires dense pixel information, whereas the range voxels are sparsely distributed. This paper proposes an approach which enables SIFT application to locate scale and rotation invariant features in 3D point clouds. The algorithm then utilizes the known point correspondence registration algorithm in order to achieve real-time registration of 3D point clouds.KeywordsPoint CloudScale Invariant Feature TransformIterative Close Point AlgorithmScale Invariant Feature Transform FeatureRANSAC AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

Visual saliency is determined through the perceptual information that enables to detect interesting regions in the query 3D models, which attracts human visual attention. Local descriptors are common and externally effective for various 3D tasks such as registration, object tracking object recognition and saliency detection. In this paper, we present a suitable solution of saliency regions-based surface patch detection in 3D point cloud images using pairwise 2D histograms of local descriptors constructed from both global and local geometry information. The local information is estimated based on the local descriptor formulated from the relation between 3D point and its neighboring points (neighbourhood points-based descriptor). The global information is estimated by extrusion shape descriptor based radial map construction. The proposed method has three well-defined steps for preprocessing workflow in order to prepare data points for salient region detection process composed of downsampling-outliers removal, normal vector computation for each point and neighbourhood determination for each point. From the experiments, the local shape descriptors adopted in this paper are reliable to detect more salient points-based surface patch detection in the 3D point cloud model. The experiments on a publicly point cloud database were attained high accuracy of the proposed surface patch detector.

Surface defect identification is a crucial task in many manufacturing systems, including automotive, aircraft, steel rolling, and precast concrete. Although image-based surface defect identification methods have been proposed, these methods usually have two limitations: images may lose partial information, such as depths of surface defects, and their precision is vulnerable to many factors, such as the inspection angle, light, color, noise, etc. Given that a three-dimensional (3D) point cloud can precisely represent the multidimensional structure of surface defects, we aim to detect and classify surface defects using a 3D point cloud. This has two major challenges: (i) the defects are often sparsely distributed over the surface, which makes their features prone to be hidden by the normal surface and (ii) different permutations and transformations of 3D point cloud may represent the same surface, so the proposed model needs to be permutation and transformation invariant. In this paper, a two-step surface defect identification approach is developed to investigate the defects’ patterns in 3D point cloud data. The proposed approach consists of an unsupervised method for defect detection and a multi-view deep learning model for defect classification, which can keep track of the features from both defective and non-defective regions. We prove that the proposed approach is invariant to different permutations and transformations. Two case studies are conducted for defect identification on the surfaces of synthetic aircraft fuselage and the real precast concrete specimen, respectively. The results show that our approach receives the best defect detection and classification accuracy compared with other benchmark methods.

Abstract. A series of panoramic images are usually used to generate a 720° panorama image. Although panoramic images are typically used for establishing tour guiding systems, in this research, we demonstrate the potential of using panoramic images acquired from multiple sites to create not only 720° panorama, but also three-dimensional (3D) point clouds and 3D indoor models. Since 3D modeling is one of the goals of this research, the location of the panoramic sites needed to be carefully planned in order to maintain a robust result for close-range photogrammetry. After the images are acquired, panoramic images are processed into 720° panoramas, and these panoramas which can be used directly as panorama guiding systems or other applications. In addition to these straightforward applications, interior orientation parameters can also be estimated while generating 720° panorama. These parameters are focal length, principle point, and lens radial distortion. The panoramic images can then be processed with closerange photogrammetry procedures to extract the exterior orientation parameters and generate 3D point clouds. In this research, VisaulSFM, a structure from motion software is used to estimate the exterior orientation, and CMVS toolkit is used to generate 3D point clouds. Next, the 3D point clouds are used as references to create building interior models. In this research, Trimble Sketchup was used to build the model, and the 3D point cloud was added to the determining of locations of building objects using plane finding procedure. In the texturing process, the panorama images are used as the data source for creating model textures. This 3D indoor model was used as an Augmented Reality model replacing a guide map or a floor plan commonly used in an on-line touring guide system. The 3D indoor model generating procedure has been utilized in two research projects: a cultural heritage site at Kinmen, and Taipei Main Station pedestrian zone guidance and navigation system. The results presented in this paper demonstrate the potential of using panoramic images to generate 3D point clouds and 3D models. However, it is currently a manual and labor-intensive process. A research is being carried out to Increase the degree of automation of these procedures.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Estimation of plant parameters is prerequisite for fast and precise management of tomato crops on commercial farm level. Destructive estimation methods may be accurate but not suitable when large farm size, spatio-temporal variability or short period of time to obtain information are considered. Therefore, this study was aimed to estimate leaf area (LA) in mature tomato plants from reconstructed geometric plant model based on 3-dimensional (3D) point cloud. A 2D laser scanner was placed on a programable linear conveyor system to move at controlled forward speed. To achieve a high density 3D point cloud of whole plant, close range (< 2 m) scanning was performed at 10 mm/s forward speed. Plant parameters capturing points per plant (PPP), plant height, plant volume, and LA were extracted from the 3D point clouds. Destructive method was utilized in laboratory for reference LA data collection, whereas other required reference data (plant height and canopy area) were measured in-situ. Linear regression models were developed and cross validated to estimate LA using variables from the 3D point clouds and reference LA. Among the models, plant height showed strong correlation with reference LA (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.90, RMSE = 40 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and followed by PPP (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.89, RMSE = 45 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), voxel grid approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.56, RMSE = 58 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and slicing approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.44, RMSE = 70 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ). Results indicate that mobile laser scanning could be a reliable non-destructive LA estimation tool for tomato and may be adapted by the commercial tomato growers for profitable management decisions.

In recent years, there has been significant growth in the ubiquity and popularity of three-dimensional (3D) point clouds, with an increasing focus on the classification of 3D point clouds. To extract richer features from point clouds, many researchers have turned their attention to various point set regions and channels within irregular point clouds. However, this approach has limited capability in attending to crucial regions of interest in 3D point clouds and may overlook valuable information from neighboring features during feature aggregation. Therefore, this paper proposes a novel 3D point cloud classification method based on global attention and adaptive graph convolution (Att-AdaptNet). The method consists of two main branches: the first branch computes attention masks for each point, while the second branch employs adaptive graph convolution to extract global features from the point set. It dynamically learns features based on point interactions, generating adaptive kernels to effectively and precisely capture diverse relationships among points from different semantic parts. Experimental results demonstrate that the proposed model achieves 93.8% in overall accuracy and 90.8% in average accuracy on the ModeNet40 dataset.

Rock mass characterization is a very important part of engineering geological investigation. For a better understanding of the rock mass behaviour, it is crucially important to obtain as much as possible information about the discontinuity network, especially about orientation and the number of dominant discontinuity sets. The traditional methodology includes field mapping which dominantly produces a limited amount of data and consequently only a rough estimate about discontinuity network. To increase the number of measurements and to eliminate orientation bias, rock mass on the Špičunak rock slope in Gorski kotar, Croatia, was analysed using a combination of 3D Point Cloud and Textured Mesh Model generated from 3D Point Cloud by Poisson surface reconstruction. Both models were obtained from Terrestrial Laser Scanning. Two considerably different parts of a rock slope, with different weathering conditions and different degrees of fracturing were mapped. Discontinuities were mapped in the field and on the models using manual mapping techniques and semi-automated methods. Manual mapping on a 3D Point Cloud and Textured Mesh Model was done by Compass plugin and by Trace a polyline tool in Cloud Compare software version V2.12 and semi-automated mapping methods were done by Discontinuity Set Extractor and qFacet Fast Marching plugin for Cloud Compare software version V2.12. This study was used to show how the application of different methodologies, for the detection of geometric properties of discontinuities, influences the result. Statistical analyses were performed on the collected data to determine differences in the accuracy between the mapping techniques. Manual mapping on the 3D Point Cloud and high-resolution Textured Mesh Model showed good agreement with field measurements, apart from the higher number of discontinuities mapped by remote sensing methods. On the other hand, significant deviations were found between manual and semi-automated mapping techniques. Semiautomated methods did not correctly detect certain discontinuities, especially bedding planes that are perpendicular to a rock face. Also, semi-automated methods overestimate the number of discontinuity sets, especially in a highly weathered and highly fractured rock mass. These differences between methods can influence kinematic analysis results. Based on the results, an appropriate methodology was proposed to utilize the advantages of both manual and semiautomated methods. The proposed approach presents a powerful tool to accurately map and measure discontinuity orientation with results comparable to the field measurements.

The inclusion of thermal infrared (TIR) data in point clouds derived from unmanned aircraft system (UAS) imagery can benefit a variety of applications in which surface temperature and 3D geometry are both important discriminants of feature type and condition. Low resolution and narrow fields of view (FOV) of current consumer-grade TIR cameras on UAS, combined with the lack of sharpness and texture in many image regions, may cause failure or poor results from structure from motion (SfM) photogrammetric software, which has gained widespread use for generating point clouds from UAS imagery. This paper proposes a photogrammetric approach for generating 3D multispectral point clouds utilizing coacquired TIR-RGB images. A 3D point cloud is first generated from the RGB imagery using standard SfM procedures. Then the TIR attributes are assigned to points, where the image coordinates of the points in TIR images are estimated using transformation parameters obtained from co-registration procedures. To obtain RGB-to-TIR transformation parameters, this study tests 3D and 2D co-registration approaches. The latter produces better results due to the challenge of calibrating the TIR camera as required for the 3D approach. This proposed approach is advantageous for generating TIR point clouds without loss of photogrammetric precision compared with solely TIR-based SfM, as the 3D accuracy, point density, and reliability are greatly enhanced.

Computing multiple aggregation levels and contextual features for road facilities recognition using mobile laser scanning data