Geometry perception and motion planning in robotic assembly based on semantic segmentation and point clouds reconstruction

Real-time motion planning and control of multi-joint robot manipulators are studied using neural networks. The proposed neural network approach consists of two modules: one for real-time collision-free motion planning, the other for real-time fine control of the robot manipulators. The motion planning module is a biologically inspired, parallel connected, topologically organised neural network, where each neuron is characterised by a shunting equation, whose state space is the robot configuration joint space. This module is capable of planning a real-time optimal path for robot manipulators through the dynamic activity landscape of the neural network. The motion control module involves of a feedforward neural network together with a traditional PD feedback loop. It is capable of achieving real-time fine motion control of robot manipulators under significant uncertainties and without any prior knowledge of the robot dynamics. This module can quickly compensate sudden changes in the robot dynamics. The real-time fine control of robot manipulators is achieved through the online learning of the neural network. Both the motion planning and motion control modules are computationally efficient, and their global stabilities are proved using Lyapunov stability theory.

Abstract. Semantic instance segmentation from scenes, serving as a crucial role for 3D modelling and scene understanding. Conducting semantic segmentation before grouping instances is adopted by the existing state-of-the-art methods. However, without additional refinement, semantic errors will fully propagate into the grouping stage, resulting in low overlap with the ground truth instance. Furthermore, the proposed methods focused on indoor level scenes, which are limited when directly applied to large-scale outdoor Airborne Laser Scanning (ALS) point clouds. Numerous instances, significant object density and scale variations make ALS point clouds distinct from indoor data. In order to address the problems, we proposed a geometric characterization-aware semantic instance segmentation network, which utilized both semantic and objectness score to select potential points for grouping. And in point cloud feature learning stage, hand-craft geometry features are taken as input for geometric characterization awareness. Moreover, to address errors propagated from previous modules after grouping, we have additionally designed a per-instance refinement module. To assess semantic instance segmentation, we conducted experiments on an open-source dataset. Additionally, we performed semantic segmentation experiments to evaluate the performance of our proposed point cloud feature learning method.

In industrial plants, piping and equipment are intricately interconnected, and there are many components with a variety of shapes. To use point clouds of industrial plants for simulation of maintenance work, it is necessary to extract the components. Deep learning is effective for recognizing components in point clouds of industrial plants. However, training classifiers is challenging due to the difficulty in acquiring diverse point cloud datasets and the labor-intensive process of annotating large-scale point clouds. A promising approach to address these issues is to train the classifier on virtual point clouds generated from CAD models. However, classifiers trained on these virtual point clouds often fail to achieve sufficient segmentation accuracy due to discrepancies between virtual point clouds and actual point clouds captured by terrestrial laser scanners. This paper proposes methods to improve segmentation accuracy by reducing these discrepancies. First, we introduce a method to incorporate features such as missing points, noise, and outliers observed in actual point clouds. Furthermore, we propose a data augmentation approach that applies up-sampling using a deep learning model trained on paired virtual and real point clouds to reduce the discrepancy between them. Our evaluation demonstrates that the proposed methods effectively improve the segmentation accuracy of point clouds of industrial plants.

Point cloud enhancement optimization and high-fidelity texture reconstruction methods for air material via fusion of 3D scanning and neural rendering

In recent years, a number of trajectory optimization algorithms have been proposed and established for motion planning of robot manipulators in complex, but static, predefined environments. To enable reactive motion planning under uncertain conditions caused, for example, by moving obstacles, this paper proposes a formulation of the trajectory optimization problem that is tailored for model predictive control. The proposed algorithmic solution leverages off-the-shelf computational tools for nonlinear model predictive control, optimization, and collision checking. In addition, a motion planning paradigm is introduced to allow for online collision-free motion when following a joystick command. The approach is validated in the context of an industrial pick-and-place application using MATLAB® and a Kinova® robot manipulator, both in simulation and with actual hardware.

Highlights A multi-view 3D camera system was built to capture depth maps and reconstruct point clouds of complete pigs in breeding farms. Geometric and morphological features were extracted from the reconstructed point clouds for measuring phenotypic traits via 3D processing algorithms. The minimal mean absolute error of predicting liveweight was 4.19 kg. The mean absolute percentage errors of predicting chest and abdominal circumferences were 2.33% and 2.06%, respectively. Abstract. Manually measuring the phenotypic traits of pigs in breeding farms can be time- and labor-intensive and cause animal stress, and subjective manual measurement errors slow down the progress of desired traits and genetic selection across multiple breeding generations. The objectives of this study were 1) developing a multi-view 3D camera system to capture depth maps and reconstruct point clouds of complete pigs in breeding farms and 2) extracting geometric and morphological features from the reconstructed point clouds for predicting phenotypic traits. The proposed 3D scanning system was installed in a swine breeding farm, and 481 PIC Camborough® pigs in parent stocks were used with RGB-depth video acquisition. Core 3D processing algorithms merged point clouds from three camera views into one complete pig point cloud, including Iterative Closest Point, Fast Point Feature Histograms, Random Sample Consensus, and Sorting. Gaussian Mixture Clustering and point cloud slicing were implemented to extract critical features for predicting six traits: abdominal circumference, chest circumference, front body height, rear body height, ear-to-tail straight length, and ear-to-tail curve length. Three standard objects were used to test the system’s performance with mean absolute errors (MAE) of 0.04 to 0.07 m. The chest and abdominal circumference correlated highly (r = 0.74) with liveweight. After comparative studies of seven combinations of the six phenotypic traits, the best weight prediction performance (MAE = 4.19 kg) was obtained with the combination of the first four traits via the XGBoost regressor. The study highlighted the innovative applications of artificial intelligence and 3D computer vision for pig breeding and genetics selection. Keywords: Artificial intelligence, Automation, Depth sensing, Expert system, Precision livestock farming.

Point cloud has emerged as the most popular three-dimensional (3D) data format in recent years for several scientific and industrial applications. Point cloud semantic segmentation has piqued the researcher’s interest, which is a crucial stage in 3D analysis and scene comprehension. Deep learning-based processing is more feasible to increase the availability of point cloud acquisition tools that is LiDAR systems at the user end. The point cloud learning achieves tremendous success in object detection, object categorization, and semantic segmentation. To summarize the recent works with chronological development, comprehensive review of projection-, voxel-, and direct point-based point cloud semantic segmentation methods is performed from various perspectives. The commonly used point cloud benchmark datasets with their characteristics are discussed, and they are used for the performance analysis and comparison of several state-of-the-art segmentation methods. The quantitative performance analysis of these deep learning models summarizes the trend of semantic segmentation of point clouds. In the context of point cloud semantic segmentation, the various methods have specific roles. Based on the review of methods working and their performance analysis, it is concluded that the projection-based methods prioritize efficiency, which is ideal in unavailability of high-performance computing system. Voxel-based methods capture overall context, serving well in 3D object classification. Point-based approaches excel in fine details and efficiency, suited for tasks like 3D semantic segmentation. Choosing the suitable method depends on the task, data, and resources. KPConv and DGCNN are popular choices, especially for precision and adaptability to point density. However, method performance varies, underlining the need for tailored selection. Hybrid approaches, combining method strengths, promise superior results.

This paper presents a framework for the motion planning and control of redundant manipulators with the added task of collision avoidance. The algorithms that were previously studied and tested by the authors for planar cases are here extended to full mobility redundant manipulators operating in a three-dimensional workspace. The control strategy consists of a combination of off-line path planning algorithms with on-line motion control. The path planning algorithm is used to generate trajectories able to avoid fixed obstacles detected before the robot starts to move; this is based on the potential fields method combined with a smoothing interpolation that exploits Bézier curves. The on-line motion control is designed to compensate for the motion of the obstacles and to avoid collisions along the kinematic chain of the manipulator; this is realized using a velocity control law based on the null space method for redundancy control. Furthermore, an additional term of the control law is introduced which takes into account the speed of the obstacles, as well as their position. In order to test the algorithms, a set of simulations are presented: the redundant collaborative robot KUKA LBR iiwa is controlled in different cases, where fixed or dynamic obstacles interfere with its motion. The simulated data show that the proposed method for the smoothing of the trajectory can give a reduction of the angular accelerations of the motors of the order of 90%, with an increase of less than 15% of the calculation time. Furthermore, the dependence of the on-line control law on the speed of the obstacle can lead to reductions in the maximum speed and acceleration of the joints of approximately 50% and 80%, respectively, without significantly increasing the computational effort that is compatible for transferability to a real system.

This paper investigates the motion planning of redundant free-floating manipulators with seven prismatic joints. On the earth, prismatic-jointed manipulators could only position their end-effectors in a desired way. However, in space, the end-effectors of free-floating manipulators can achieve both the desired orientation and desired position due to the dynamical coupling between manipulator and satellite movement, which is formally expressed by linear and angular momentum conservation laws. In this study, a tractable algorithm particle swarm optimization combined with differential evolution (PSODE) is provided to deal with the motion planning of redundant free-floating prismatic-jointed manipulators, which could avoid the pseudo inverse of the Jacobian matrix. The polynomial functions, as argument in sine functions are used to specify the joint paths. The coefficients of the polynomials are optimized to achieve the desired end-effector orientation and position, and simultaneously minimize the unit-mass-kinetic energy using the redundancy. Relevant simulations prove that this method provides satisfactory smooth paths for redundant free-floating prismatic-jointed manipulators. This study could help to recognize the advantages of redundant prismatic-jointed space manipulators.

Objectives. The work analyzes features of one of the most promising approaches to solve the problems for motion planning of autonomous robotic manipulators of various types and purposes using the rapidly exploring random tree (RRT) method. The development of modern robotics is shown to be inextricably linked with the improvement of the designs of the created samples, for which the placement of a manipulator on platform becomes a typical layout option. Prospects for using the RRT method as a constructive basis for creating a universal motion planner are evaluated for mobile and robotic manipulators, including autonomous robotic systems with a manipulator on a moving platform.Methods. The object of the research is the RRT method and its well-known modifications RRT* and RRT-Connect. The effectiveness of applying such methods for solving problems associated with planning the motions of robotic manipulators of various types was evaluated using computer and natural simulation methods.Results. Based on a review of the literature and the results of the research, the wide possibilities of the RRT method can be used for solving motion planning problems not only for mobile and robotic manipulators, but also for robotic systems on whose transport platform an onboard manipulator has been installed (including those having a redundant or reconfigurable structure). The effectiveness of the applied application of the RRT method is confirmed by examples of modeling a mobile platform with an onboard manipulator and the results of full-scale experiments with a prototype of the ARAKS reconfigurable mechatronic-modular robotic manipulators (RTU MIREA, Russia). It can be experimentally demonstrated and theoretically substantiated that the final dimension of the exploring tree, and hence the time of its construction up to reaching a given target state, is largely determined by the value of the growth factor.Conclusions. The generalization of the results obtained opens up real prospects for using the RRT method as a constructive basis not only for creating universal means for motion planning mobile robotic systems with an onboard manipulator, but also for solving the problems of automating the docking of autonomous mobile platforms.

Stability analysis and motion planning strategies with fuzzy coordinator and learning for mobile base robotic part macro/micro-assembly tasks

LESA-Net: Semantic segmentation of multi-type road point clouds in complex agroforestry environment

Point cloud semantic segmentation in urban scenes plays a vital role in intelligent city modeling, autonomous driving, and urban planning. Point cloud semantic segmentation based on deep learning methods has achieved significant improvement. However, it is also challenging for accurate semantic segmentation in large scenes due to complex elements, variety of scene classes, occlusions, and noise. Besides, most methods need to split the original point cloud into multiple blocks before processing and cannot directly deal with the point clouds on a large scale. We propose a novel context-aware network (CAN) that can directly deal with large-scale point clouds. In the proposed network, a Local Feature Aggregation Module (LFAM) is designed to preserve rich geometric details in the raw point cloud and reduce the information loss during feature extraction. Then, in combination with a Global Context Aggregation Module (GCAM), capture long-range dependencies to enhance the network feature representation and suppress the noise. Finally, a Context-Aware Upsampling Module (CAUM) is embedded into the proposed network to capture the global perception from a broad perspective. The ensemble of low-level and high-level features facilitates the effectiveness and efficiency of 3D point cloud feature refinement. Comprehensive experiments were carried out on three large-scale point cloud datasets in both outdoor and indoor environments to evaluate the performance of the proposed network. The results show that the proposed method outperformed the state-of-the-art representative semantic segmentation networks, and the overall accuracy (OA) of Tongji-3D, Semantic3D, and S3DIS is 96.01%, 95.0%, and 88.55%, respectively.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

Semantic segmentation in the context of 3D point clouds for the railway environment holds a significant economic value, but its development is severely hindered by the lack of suitable and specific datasets. Additionally, the models trained on existing urban road point cloud datasets demonstrate poor generalisation on railway data due to a large domain gap caused by non‐overlapping special/rare categories, for example, rail track, track bed etc. To harness the potential of supervised learning methods in the domain of 3D railway semantic segmentation, we introduce RailPC, a new point cloud benchmark. RailPC provides a large‐scale dataset with rich annotations for semantic segmentation in the railway environment. Notably, RailPC contains twice the number of annotated points compared to the largest available mobile laser scanning (MLS) point cloud dataset and is the first railway‐specific 3D dataset for semantic segmentation. It covers a total of nearly 25 km railway in two different scenes (urban and mountain), with 3 billion points that are finely labelled as 16 most typical classes with respect to railway, and the data acquisition process is completed in China by MLS systems. Through extensive experimentation, we evaluate the performance of advanced scene understanding methods on the annotated dataset and present a synthetic analysis of semantic segmentation results. Based on our findings, we establish some critical challenges towards railway‐scale point cloud semantic segmentation. The dataset is available at https://github.com/NNU‐GISA/GISA‐RailPC, and we will continuously update it based on community feedback.