Case Study: Improving the Quality of Dairy Cow Reconstruction with a Deep Learning-Based Framework

In this work, the problem of reliably checking collisions between robot manipulators and the surrounding environment in short time for tasks, such as replanning and object grasping in clutter, is addressed. Geometric approaches are usually applied in this context; however, they can result not suitable in highly time-constrained applications. The purpose of this paper is to present a learning-based method able to outperform geometric approaches in clutter. The proposed approach uses a neural network (NN) to detect collisions online by performing a classification task on the input represented by the depth image or point cloud containing the robot gripper projected into the application scene. Specifically, several state-of-the-art NN architectures are considered, along with some customization to tackle the problem at hand. These approaches are compared to identify the model that achieves the highest accuracy while containing the computational burden. The analysis shows the feasibility of the robot collision checker based on a deep learning approach. In fact, such approach presents a low collision detection time, of the order of milliseconds on the selected hardware, with acceptable accuracy. Furthermore, the computational burden is compared with state-of-the-art geometric techniques. The entire work is based on an industrial case study involving a KUKA Agilus industrial robot manipulator at the Technology $\&$ Innovation Center of KUKA Deutschland GmbH, Germany. Further validation is performed with the Amazon Robotic Manipulation Benchmark (ARMBench) dataset as well, in order to corroborate the reported findings.

In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized probability density, thereby moving sampled points toward the high-likelihood regions. Our model directly predicts the gradient of the log density field and can be trained with a simple objective adapted from score-based generative models. We show that our method can reach state-of-the-art performance for point cloud auto-encoding and generation, while also allowing for extraction of a high-quality implicit surface. Code is available at https://github.com/RuojinCai/ShapeGF .

In the field of 3D point cloud generation, numerous 3D generative models have demonstrated the ability to generate diverse and realistic 3D shapes. However, the majority of these approaches struggle to generate controllable 3D point cloud shapes that meet user-specific requirements, hindering the large-scale application of 3D point cloud generation. To address the challenge of lacking control in 3D point cloud generation, we are the first to propose controlling the generation of point clouds by shape structures that comprise part existences and part adjacency relationships. We manually annotate the adjacency relationships between the segmented parts of point cloud shapes, thereby constructing a StructureGraph representation. Based on this StructureGraph representation, we introduce StrucADT, a novel structure-controllable point cloud generation model, which consists of StructureGraphNet module to extract structure-aware latent features, cCNF Prior module to learn the distribution of the latent features controlled by the part adjacency, and Diffusion Transformer module conditioned on the latent features and part adjacency to generate structure-consistent point cloud shapes. Experimental results demonstrate that our structure-controllable 3D point cloud generation method produces high-quality and diverse point cloud shapes, enabling the generation of controllable point clouds based on user-specified shape structures and achieving state-of-the-art performance in controllable point cloud generation on the ShapeNet dataset.

Point cloud data in the real world is often affected by occlusion and light reflection, leading to incompleteness of the data. Large‐region missing point clouds will cause great deviations in downstream tasks. A dual feature fusion network (DFF‐Net) is proposed to improve the accuracy of the completion of a large missing region of the point cloud. First, a dual feature encoder is designed to extract and fuse the global and local features of the input point cloud. Subsequently, a decoder is used to directly generate a point cloud of missing region that retains local details. In order to make the generated point cloud more detailed, a loss function with multiple terms is employed to emphasise the distribution density and visual quality of the generated point cloud. A large number of experiments show that the authors’ DFF‐Net is better than the previous state‐of‐the‐art methods in the aspect of point cloud completion.

Unmanned aerial vehicles (UAVs) or drones are lightly weighted platforms with numerous advantages and can acquire very high-resolution data rapidly with less expenditure. However, the processing of very high-resolution data acquired through the sensors mounted on the UAV platform is a major challenge, and the commercial UAV data processing software is expensive. These software packages require high-configuration hardware and follow the general workflow of conventional photogrammetric approach for image alignment, point cloud generation, mesh generation, DSM, and ortho-mosaic generation. The integration of computer vision algorithms into digital photogrammetric techniques has made it possible to process the high-resolution datasets quickly. A wide variety of open-source software utilities are available that can perform these tasks individually; however, an integrated solution is always desirable. An attempt is made to develop an open-source tool for photogrammetric processing of UAV data using open-source libraries in Python. The software provides a complete workflow for photogrammetric processing of UAV data ranging from camera calibration, point cloud generation from selective frames based on B/H ratio, direct georeferencing of generated point cloud and digital elevation model (DEM) generation. The developed software is compatible with both 32-bit and 64-bit operating systems and low configuration hardware.

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications [1,2].METHODS & RESULTS: While new approaches for deep neural networks are rapidly developing [1], the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ [3], can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments [1], purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo [5], enables to perform ML for change analysis in multitemporal point clouds [6]. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo). REFERENCES:[1] Esmorís Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.[2] Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.112772 [3] HELIOS++: https://github.com/3dgeo-heidelberg/helios[4] VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d[5] py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo[6] Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034

As a mainstream form of 3D data, point clouds are widely used in computer vision for tasks such as segmentation, classification, and target detection due to their simple representation method and high stability and accuracy. Considering the issues of noise points and uneven point distribution in current generation models, we propose a novel adversarial network model, SAC-GAN, which incorporates both a self-attention mechanism and a curvature learning mechanism. Firstly, the feature enhancement module and the pre-processing module, which are based on the ShapeNetCore open dataset, are designed on the generator to enhance the authenticity of local geometric details in the generated point cloud. Secondly, the loss function of the discriminator is adjusted to combine the traditional Wasserstein distance with the normal vector of key points to guide the generation of subtle features of point clouds and improve the quality and consistency of generated point clouds; Finally, to enhance the discriminator's capacity to extract both local and global features, a self-attention mechanism is introduced. This enhances the discriminator's capacity to discern the details of the generated point cloud and offers superior feedback to the generator. Experimental results indicate that the proposed point cloud generation model outperforms existing methods, including TreeGAN, SP-GAN, PDGN, and WarpingGAN, in terms of generation quality. Specifically, the model achieves a reduction in JSD by 4.24, a decrease in MMD by 0.8, and an increase in COV by 1.25%. It can be proven that the point cloud generated by SAC-GAN model has a good performance in shape integrity and authenticity.

This paper presents a novel approach to construct spatially-referenced, multidimensional virtual city models from remotely generated point clouds for areas that lack reliable geographical reference data. A multidimensional point cloud is an unstructured array of single, irregular points in a spatial 3D coordinate system plus time stamp. If geospatial reference points are available, a point cloud is geo-referenced. Geo-referenced point clouds contain a high-precision reference dataset. Point clouds can be utilised in a variety of applications. They are particularly suitable for the representation of surfaces, structures, terrain and objects. Point clouds are used here to generate a virtual 3D city model representing the complex, granular cityscape of Jerusalem and its centre, the Old City. The generation of point clouds is based on two data acquisition methods: active data capture by laser scanning and passive data collection by photogrammetric methods. In our case, very high-resolution stereo imagery in visible light and near infrared bands have been systematically acquired an aerial flight campaign. The spatio-temporal data gathered necessitate further processing to extract the geographical reference and semantic features required in a specific resolution and scale. An insight is given into the processing of an unstructured point cloud to extract and classify the 3D urban fabric and reconstruct its objects. Eventually, customised, precise and up-to-date geographical datasets can be made available for a defined region at a defined resolution and scale.

The permanent terrestrial laser scanning (PLS) system has opened the possibilities for efficient data acquisition with high-temporal and spatial resolution, thus allowing for improved capture and analyses of complex geomorphological changes on the Earth's surface. Accurate georeferencing of generated four-dimensional point clouds (4DPC) from PLS is the prerequisite of the following change analysis. Due to the massive data volume and potential changes between scans, however, efficient, robust, and automatic georeferencing of 4DPC remains challenging, especially in scenarios lacking signalized and reliable targets. This georeferencing procedure can be typically realized by designating a reference epoch and registering all other scans to this epoch. Addressing the challenges in targetless registration of topographic 4DPC, we propose a simple and efficient registration method called Piecewise-ICP, which first segments point clouds into piecewise patches and aligns them in a piecewise manner.Assuming the stable areas on monitored surfaces are locally planar, supervoxel-based segmentation is employed to generate small planes from adjacent point clouds. These planes are then refined and classified by comparing defined correspondence distances to a monotonically decreasing distance threshold, thus progressively eliminating unstable planes in an efficient iterative process as well as preventing local minimization in the ICP process. Finally, point-to-plane ICP is performed on the centroids of the remaining stable planes. We introduce the level of detection in change analysis to determine the minimum distance threshold, which mitigates the influence of outliers and deformed areas on registration accuracy. Besides, the spatial distribution of empirical registration uncertainties on registered point clouds is derived based on the variance-covariance propagation law.Our registration method is demonstrated on two datasets: (1) Synthetic point cloud time series with defined changes and transformation parameters, and (2) a 4DPC dataset from a PLS system installed in the Vals Valley (Tyrol, Austria) for monitoring a rockfall. The experimental results show that the proposed algorithm exhibits higher registration accuracy compared to the existing robust ICP variants. The real-time capability of Piecewise-ICP is significantly improved owing to the centroid-based point-to-plane ICP and the efficient iteration process.

Efficient and reliable grasp pose generation plays a crucial role in robotic manipulation tasks. The advancement of deep learning techniques applied to point cloud data has led to rapid progress in grasp detection. However, point cloud data has limitations: no appearance information and susceptibility to sensor noise. In contrast, color Red, Green, Blue (RGB) images offer high-resolution and intricate textural details, making them a valuable complement to the 3-D geometry offered by point clouds or depth (D) images. Nevertheless, the effective integration of appearance information to enhance point cloud-based grasp detection remains an open question. In this study, we extend the concepts of VoteGrasp <xref ref-type="bibr" rid="ref1">[1]</xref> and introduce an innovative deep learning approach referred to as VoteGrasp Red, Green, Blue, Depth (RGBD). To build robustness to occlusion, the proposed model generates candidates by casting votes and accumulating evidence for feasible grasp configurations. This methodology revolves around fuzing votes extracted from images and point clouds. To further enhance the collaborative effect of merging appearance and geometry features, we introduce a context learning module. We exploit contextual information by encoding the dependency of objects in the scene into features to boost the performance of grasp generation. The contextual information enables our model to increase the likelihood that the generated grasps are collision-free. The efficacy of our model is verified through comprehensive evaluations on the demanding GraspNet-1Billion dataset, leading to a significant improvement of 9.3 in average precision (AP) over the existing state-of-the-art results. Additionally, we provide extensive analyses through ablation studies to elucidate the contributions of each design decision.

The pose (position and attitude) measurement of space non-cooperative targets is one of the key technologies in on-orbit service. Traditional stereo-vision-based methods are easily affected by illumination conditions, while the 3D laser radar has a good stability, but the point cloud is sparse. In this paper, a method based on the 3D laser radar and stereo-vision information is proposed to measure the pose and estimate the motion for non-cooperative targets. The proposed method can not only adapt to the harsh lighting conditions, but also has better robustness and higher precision. First, find the matching point corresponding to the binocular camera by using the stereo matching algorithm, and then, calculate the 3D coordinates of the matching points according to the least squares method. Combined with the fusion calibration result, the point cloud data acquired by the 3D laser radar is converted to the left camera frame. Second, integrate the point cloud data obtained by the above two sensors to obtain a complete point cloud model of the non-cooperative target. Third, the obtained point cloud model is registered, and the rigid transformation matrix of two adjacent point clouds is solved. Then, the relative pose and velocity (linear velocity and angular velocity) of the non-cooperative target are estimated by the extended Kalman filter (EKF) algorithm. Finally, an experimental system was built. It consists of a UR5 robot, a satellite mockup, two cameras, a 3D laser radar, and an API laser tracker (for evaluating the accuracy of measurement), was built. The experiment results verified the proposed method.

Depth cameras started to gain popularity in agricultural applications during the last years. This type of cameras has been implemented mainly for three-dimensional (3D) reconstruction of objects in indoor or outdoor sceneries. The use of such cameras in the construction of 3D models for simulation purposes of complex structures that are usually met in nature, such as trees and other plants, is a great challenge. Remarkably, agricultural environments are extremely complex. Thus, the proper setup and implementation of such technologies is particularly important in order to attain useable data. So far, the depth information collected using these cameras varies among different objects’ structure and sensing conditions due to the uncertainty of the outdoor environment. The use of a specific methodology using color and depth images gives the opportunity to extract geometrical characteristics information about point clouds of the targeted objects. This chapter explores the different technologies used by depth cameras and presents several applications concerning indoor and outdoor environments by presenting indicative scenarios for agricultural applications. Towards that direction, a 3D reconstruction of trees was established producing point clouds from Red Green Blue Depth (RGB-D) images acquired in real field conditions. The point cloud samples of trees were collected using an unmanned ground vehicle (UGV) and imported in Gazebo in order to visualize a simulation of the environment. This simulation technique can be used for testing and evaluating the navigation of robotic systems. By further analyzing the resulted 3D point clouds, various geometrical measurements of the simulated samples, such as the volume or the height of tree canopies, can be calculated. Possible weaknesses of this procedure are mainly attributed to the camera’s limitations and the sampling parameters. However, results show that it is possible to establish a suitable simulation environment to implement it in several agricultural applications by utilizing automated unmanned robotic platforms.KeywordsDepth camerasUnmanned ground vehiclesScene reconstruction3D point cloudAgricultural robotsSituation awareness

The aim of the presentexperiment was to evaluate the influence of I 500 TU injection of Pregnant Mare Serum Gonadotrophin (PMSG) at the mating program in the Fresian Holstein (FH) dairy cow on the performancesof milk production, quality, and production efficiency. The experimentwas performed at Banaran Village, Cepogo Subdistrict ofBoyolali District used twelve heifers and the-pregnant first parity cow of about 350-450 kg body weight. A split-plot experimental design with heifer (Pol and first parity (P1) as a main plot, and PMSG doses (T0 and T ,)as a sub plot. Thus the treatmentwere I) heifer with no PSMG injection (P0T0), 2) PMSG injected heifer (P0T,), 3) first parity cow with no PMSG injection, and4) PMSG injected first parity cow P,T,). Measurements were conducted on 90 days milk production, mass weight of milk, fat and lactose contents, and milk production efficiency. Results showed that average milk productions of not injected (804.67 l) and injected PMSG heifers ( 1,286.92 l) were significantly different (P 0.05) influenced average milk productions of first parity cows ( J.0491 versus 1,4832.52 J, for not injected and injected PMSG cow, respectively). Fat production of the not injected and injected PMSG heifers were 25.27 and 41.12 kg (P 0.05). Milk production efficiency of not injected and injected PMSG heifers were 11.13 and 18.45%, respectively (P 0.05). In general there were no significant differenton theeffectof PMSG injection on milk quality components (mass weight, fat and lactose contents). In conclusion, in the present experiment PMSG injection improved milk pro• duction and it efficiency, but did not improved milk quality.

The direct extension of 2D image learning to three-dimensional space is 3D point cloud learning. Recently, point cloud learning has shown significant results in 3D shape estimation and semantic segmentation. Despite these advancements, fundamental problems in point cloud learning still pose significant challenges. These problems include representation learning, shape generation, shape segmentation, and shape matching. In this paper, we propose a cognitive self-attention based learning approach to aggregate global representation of 3D shapes from point cloud data. We also integrate 3D point data with a binary tree structure to build a point cloud generator. We further design a novel Generative Adversarial Network (GAN) architecture to generate point clouds resembling the ground truth that could be used for unsupervised learning of 3D shapes. Relying on a self-attention mechanism, our framework, called SAPCGAN, aggregates the global graph features to correct the structural information of 3D shapes in the latent space. Finally, we compare the performance of two gradient penalty methods used in stabilizing the training of our GAN system. We show that our framework has high training efficiency and can provide state-of-the-art results in 3D point cloud generation. The performance of our is demonstrated with both quantitative and qualitative experimental evaluations. Furthermore, the generated 3D point clouds can be segmented into their natural semantic parts, such as, for example the four legs of a chair, the wings of an air plane, or the four wheels of a car.

This paper introduces at first GQS (General Quadric Surface) equation, then considers that utilizing depth images referenced to various datum planes to model point cloud. After getting reference datum, this paper proposing the methods of creating depth images from point cloud and the specific steps with reference to different datums. Finally we choose some point cloud data of ancient building components from laser scanning data of Forbidden City in China as experiment data, and develop a depth images modeling system. Experiment results demonstrate the stability and high precision of the method of plane, cylinder and sphere fitting as well as the validity of depth images to represent point cloud of object.