Augmented LiDAR Simulator for Autonomous Driving

With the advancement of industrial automation, the frequency of human–robot interaction (HRI) has significantly increased, necessitating a paramount focus on ensuring human safety throughout this process. This paper proposes a simulation‐assisted neural network for point cloud segmentation in HRI, specifically distinguishing humans from various surrounding objects. During HRI, readily accessible prior information, such as the positions of background objects and the robot's posture, can generate a simulated point cloud and assist in point cloud segmentation. The simulation‐assisted neural network utilizes simulated and actual point clouds as dual inputs. A simulation‐assisted edge convolution module in the network facilitates the combination of features from the actual and simulated point clouds, updating the features of the actual point cloud to incorporate simulation information. Experiments of point cloud segmentation in industrial environments verify the efficacy of the proposed method.

Lidar point cloud distortion from moving object is an important problem in autonomous driving, and recently becomes more demanding with the emerging of oscillating type lidars, which feature back-and-forth scanning patterns and complex distortions. Accurately correcting the point cloud distortion would not only describe the 3D moving objects more accurately, but also enable accurate estimation of moving objects’ velocities with enhanced prediction and tracking capabilities. A lidar and camera fusion approach is proposed to correct the oscillating lidar distortions with full velocity estimation. Lidar measures the time-of-flight distance accurately in the radial direction but only with sparse angular information while camera as a complementary sensor could provide a dense angular resolution. In addition, the proposed framework utilizes a probabilistic Kalman-filter approach to combine the estimated velocities and track the moving objects with their real-time velocities and correct point clouds. The proposed framework is evaluated on real road data and consistently outperforms other methods. The complete framework is open-sourced to accelerate the adoption of the emerging lidars.

The genomic prediction of phenotypes and breeding values in animals and plants has developed rapidly into its own research field. Results of genomic prediction studies are often difficult to compare because data simulation varies, real or simulated data are not fully described, and not all relevant results are reported. In addition, some new methods have been compared only in limited genetic architectures, leading to potentially misleading conclusions. In this article we review simulation procedures, discuss validation and reporting of results, and apply benchmark procedures for a variety of genomic prediction methods in simulated and real example data. Plant and animal breeding programs are being transformed by the use of genomic data, which are becoming widely available and cost-effective to predict genetic merit. A large number of genomic prediction studies have been published using both simulated and real data. The relative novelty of this area of research has made the development of scientific conventions difficult with regard to description of the real data, simulation of genomes, validation and reporting of results, and forward in time methods. In this review article we discuss the generation of simulated genotype and phenotype data, using approaches such as the coalescent and forward in time simulation. We outline ways to validate simulated data and genomic prediction results, including cross-validation. The accuracy and bias of genomic prediction are highlighted as performance indicators that should be reported. We suggest that a measure of relatedness between the reference and validation individuals be reported, as its impact on the accuracy of genomic prediction is substantial. A large number of methods were compared in example simulated and real (pine and wheat) data sets, all of which are publicly available. In our limited simulations, most methods performed similarly in traits with a large number of quantitative trait loci (QTL), whereas in traits with fewer QTL variable selection did have some advantages. In the real data sets examined here all methods had very similar accuracies. We conclude that no single method can serve as a benchmark for genomic prediction. We recommend comparing accuracy and bias of new methods to results from genomic best linear prediction and a variable selection approach (e.g., BayesB), because, together, these methods are appropriate for a range of genetic architectures. An accompanying article in this issue provides a comprehensive review of genomic prediction methods and discusses a selection of topics related to application of genomic prediction in plants and animals.

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications [1,2].METHODS & RESULTS: While new approaches for deep neural networks are rapidly developing [1], the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ [3], can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments [1], purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo [5], enables to perform ML for change analysis in multitemporal point clouds [6]. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo). REFERENCES:[1] Esmorís Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.[2] Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.112772 [3] HELIOS++: https://github.com/3dgeo-heidelberg/helios[4] VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d[5] py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo[6] Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034

Automated driving is expected to enormously evolve the transportation industry and ecosystems. Advancement in communications and sensor technologies have further accelerated the realization process of the autonomous driving goals. There are a number of autonomous driving initiatives around the world with varying objectives and scope, e.g. vehicle perception in a controlled environment or highway settings. Autonomous driving in a more complex environment with mixed traffic poses major challenges. The solutions for such environments is the focus of this paper. We start with a quick overview of current autonomous driving development activities worldwide. We then discuss the solution concept for autonomous driving in urban environments and its enabling components, e.g. road digitization and flexible communication infrastructure, to realize an urban autonomous driving testbed. We highlight the major challenges hindering the realization use-cases of Level 5 autonomous driving. Solution sketches to address these or similar changes are briefly discussed. We also implement some elements of the solution approaches on the real test-road. We demonstrate an artificial intelligence based approach for the analysis of real traffic data measured on the testbed. We implement approaches for predicting the network resource demands and allocation, which are crucial for realizing the use-cases of autonomous driving in complex environments. For the experiments, real data from the test-road is used. Results show that traffic patterns and resource demands are predicted accurately. These experiments are expected to instrumental for realizing other use-cases of autonomous driving.

Optimization of an independent component analysis approach for artifact identification and removal in magnetoencephalographic signals.

For the problem that it is difficult to effectively cluster lidar point clouds with irregular shapes and uneven densities, a Neighborhood Effective Line Density (NELD)-based Euclidean Clustering (NELD-EC) algorithm is proposed in this paper. The NELD-EC algorithm first eliminates the interfering points within the neighborhood of the data point by utilizing the distance relationship and calculates the NELD of the data point using the effective neighborhood set without interfering points of the data point. The NELD of a data point is taken as the local density of that data point. Then, the NELD-EC algorithm conducts clustering processing using the NELD of all data points and uses the reciprocal of the harmonic average of the local densities of all data points within each cluster after clustering as the distance threshold for the data points within the cluster. Finally, the NELD-EC algorithm completes the clustering of the point cloud based on the adjusted adaptive distance threshold. The clustering experimental results on simulated point clouds, fixed point clouds, and sequential point clouds indicate that, compared with several other typical Euclidean clustering algorithms, the NELD-EC algorithm requires simpler parameters to be set, is less sensitive to the initial distance threshold, can effectively reduce the occurrence probabilities of over-segmentation and under-segmentation, and has strong stability in clustering performance. The NELD-EC algorithm is more suitable for processing sequential point clouds in actual dynamic and complex scenarios.

Lidar has received a lot of attention due to its precise ranging accuracy. Ground points filtering is an important task in point clouds processing. It’s a challenge to model the ground surface and filter the point clouds accurately in the case of complex ground undulations, occlusions, and sparse point clouds. A novel ground surface modeling method based on a hybrid regression technique is proposed in this paper. The method integrates Gaussian process regression (GPR) and robust locally weighted regression (RLWR) by dividing the point clouds that are projected on the polar grid map into radial and circumferential filtering processes to form a hybrid regression model, which has the ability to eliminate the influence of outliers and model the ground surface robustly. First, the RLWR combined with gradient filter is applied to fit the sampled points in the radial direction, which will exclude outliers and get the fitting ground line. All radial fitting lines constitute the seed skeleton of the whole plane. Then, based on the seeds in the same circumferential of the skeleton, the GPR is applied to construct the ground surface model. The comparative experiments are implemented quantitatively and qualitatively on the simulated point clouds and measured data. The results show that the proposed method performs well in most real scenarios, even in the cases of ground undulation, occlusion, and sparse point clouds.

Three-dimensional information perception from point clouds is of vital importance for improving the ability of machines to understand the world, especially for autonomous driving and unmanned aerial vehicles. Data annotation for point clouds is one of the most challenging and costly tasks. In this paper, we propose a closed-loop and virtual–real interactive point cloud generation and model-upgrading framework called Parallel Point Clouds (PPCs). To our best knowledge, this is the first time that the training model has been changed from an open-loop to a closed-loop mechanism. The feedback from the evaluation results is used to update the training dataset, benefiting from the flexibility of artificial scenes. Under the framework, a point-based LiDAR simulation model is proposed, which greatly simplifies the scanning operation. Besides, a group-based placing method is put forward to integrate hybrid point clouds, via locating candidate positions for virtual objects in real scenes. Taking advantage of the CAD models and mobile LiDAR devices, two hybrid point cloud datasets, i.e., ShapeKITTI and MobilePointClouds, are built for 3D detection tasks. With almost zero labor cost on data annotation for newly added objects, the models (PointPillars) trained with ShapeKITTI and MobilePointClouds achieved 78.6% and 60.0% of the average precision of the model trained with real data on 3D detection, respectively.

Based on benchmarking various methods and analyzing multiple real scRNA-seq datasets, a computational platform/workflow and a set of tips for best practices are developed for analyzing causal interactions or relationships in single cells.

Paris-CARLA-3D is a dataset of several dense colored point clouds of outdoor environments built by a mobile LiDAR and camera system. The data are composed of two sets with synthetic data from the open source CARLA simulator (700 million points) and real data acquired in the city of Paris (60 million points), hence the name Paris-CARLA-3D. One of the advantages of this dataset is to have simulated the same LiDAR and camera platform in the open source CARLA simulator as the one used to produce the real data. In addition, manual annotation of the classes using the semantic tags of CARLA was performed on the real data, allowing the testing of transfer methods from the synthetic to the real data. The objective of this dataset is to provide a challenging dataset to evaluate and improve methods on difficult vision tasks for the 3D mapping of outdoor environments: semantic segmentation, instance segmentation, and scene completion. For each task, we describe the evaluation protocol as well as the experiments carried out to establish a baseline.

XLPE cable joint defects measurement method based on point cloud remapping

Point cloud data generated by laser scanners often comes in a very large file size, as it carries highly dense information of the scanned area. One of the ways to control and manage this huge data size is during the data collection process, where it can be conducted in smaller area of scanning, which can reduce the file size, and this is appreciated especially when dealing with bigger scale projects like building or life size monuments scanning. However, adapting this approach would require suitable registration methods in order to integrate and match the data together before further processing and reconstruction. This paper highlights on an automatic method of geometrical-based point cloud data registration, which work best when the scanning area are collected in half. In here, the geometrical entity representing both point clouds that need to be registered (i.e. straight surface) will be detected, and angle between the differences of these surfaces will be calculated before the second point cloud can be adjusted accordingly to be matched and registered with the first point cloud data. Experimental results on real point cloud data representing building interior shows that the algorithm works well in registering point cloud data together, hence making the process of handling huge point cloud data to be more manageable.

The 3D object detection is becoming indispensable for environmental perception in autonomous driving. Light detection and ranging (LiDAR) point clouds often fail to distinguish objects with similar structures and are quite sparse for distant or small objects, thereby introducing false and missed detections. To address these issues, LiDAR is often fused with cameras due to the rich textural information provided by images. However, current fusion methods suffer the inefficient data representation and inaccurate alignment of heterogeneous features, leading to poor precision and low efficiency. To this end, we propose a plug-and-play module termed range-image fusion (RI-Fusion) to achieve an effective fusion of LiDAR and camera data, designed to be easily accessible by existing mainstream LiDAR-based algorithms. In this process, we design an image and point cloud alignment method by converting a point cloud into a compact range-view representation through a spherical coordinate transformation. The range image is then integrated with a corresponding camera image utilizing an attention mechanism. The original range image is then concatenated with fusion features to retain point cloud information, and the results are projected onto a spatial point cloud. Finally, the feature-enhanced point cloud can be input into a LiDAR-based 3D object detector. The results of validation experiments involving the KITTI 3D object detection benchmark showed that our proposed fusion method significantly enhanced multiple mainstream LiDAR-based 3D object detectors, PointPillars, SECOND, and Part <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{A}{^{2}}$ </tex-math></inline-formula> , improving the 3D mAP (mean Average Precision) by 3.61%, 2.98%, and 1.27%, respectively, particularly for small objects such as pedestrians and cyclists.

In the recent era, a lot of interest is attracted by the autonomous vehicles which can sense surroundings and navigate without human intervention. Object detection and recognition form a major part of autonomous driving systems. Lidar sensors can be used to capture point clouds of driving environment. Detecting multiple 3D objects in point clouds in real time and defining their boundaries with the help of 3D bounding boxes are critical in motion planning by self driving systems. This paper proposes a LiDAR-based 3D object detection system that operates in real-time, with emphasis on autonomous driving scenarios. A state-of-the-art 2D standard object detector for RGB images, YOLOv4, is used as the base for object detection. The multi-class 3D bounding boxes are generated using a complex regression approach. An Euler-Region-Proposal Network (E-RPN) is used to predict the pose of the object. The proposed model receives point cloud data as input and outputs 3D bounding boxes with classes in real-time. The experiments done on the KITTI benchmark dataset proves that the proposed system outperforms existing methods in terms of accuracy and performance.