Human Point Cloud Inpainting for Industrial Human-Robot Collaboration Using Deep Generative Model

In this paper, we propose a novel gait recognition framework - Spherical Space Model with Human Point Clouds (SSM-HPC) to recognize front view of human gait. A new gait representation - Marching in Place (MIP) gait is also introduced which preserves the spatiotemporal characteristics of individual gait manner. In comparison with the previous studies on gait recognition which usually use human silhouette images from image sequences, this research applies three dimensional (3D) point clouds data of human body obtained from stereo camera. The proposed framework exhibits gait recognition rates superior to those of other gait recognition methods.

In the recent era, a lot of interest is attracted by the autonomous vehicles which can sense surroundings and navigate without human intervention. Object detection and recognition form a major part of autonomous driving systems. Lidar sensors can be used to capture point clouds of driving environment. Detecting multiple 3D objects in point clouds in real time and defining their boundaries with the help of 3D bounding boxes are critical in motion planning by self driving systems. This paper proposes a LiDAR-based 3D object detection system that operates in real-time, with emphasis on autonomous driving scenarios. A state-of-the-art 2D standard object detector for RGB images, YOLOv4, is used as the base for object detection. The multi-class 3D bounding boxes are generated using a complex regression approach. An Euler-Region-Proposal Network (E-RPN) is used to predict the pose of the object. The proposed model receives point cloud data as input and outputs 3D bounding boxes with classes in real-time. The experiments done on the KITTI benchmark dataset proves that the proposed system outperforms existing methods in terms of accuracy and performance.

Current point cloud extraction methods based on photogrammetry generate large amounts of spurious detections that hamper useful 3D mesh reconstructions or, even worse, the possibility of adequate measurements. Moreover, noise removal methods for point clouds are complex, slow and incapable to cope with semantic noise. In this work, we present body2vec, a model-based body segmentation tool that uses a specifically trained Neural Network architecture. Body2vec is capable to perform human body point cloud reconstruction from videos taken on hand-held devices (smartphones or tablets), achieving high quality anthropometric measurements. The main contribution of the proposed workflow is to perform a background removal step, thus avoiding the spurious points generation that is usual in photogrammetric reconstruction. A group of 60 persons were taped with a smartphone, and the corresponding point clouds were obtained automatically with standard photogrammetric methods. We used as a 3D silver standard the clean meshes obtained at the same time with LiDAR sensors post-processed and noise-filtered by expert anthropological biologists. Finally, we used as gold standard anthropometric measurements of the waist and hip of the same people, taken by expert anthropometrists. Applying our method to the raw videos significantly enhanced the quality of the results of the point cloud as compared with the LiDAR-based mesh, and of the anthropometric measurements as compared with the actual hip and waist perimeter measured by the anthropometrists. In both contexts, the resulting quality of body2vec is equivalent to the LiDAR reconstruction.

The recent activity in the area of autonomous vehicle navigation has initiated a series of reactions that stirred the automobile industry, pushing for the fast commercialization of this technology which, until recently, seemed futuristic. The LiDAR sensor is able to provide a detailed understanding of the environment surrounding the vehicle making it useful in a plethora of autonomous driving scenarios. Segmenting the 3D point cloud that is provided by modern LiDAR sensors, is the first important step towards the situational assessment pipeline that aims for the safety of the passengers. This step needs to provide accurate segmentation of the ground surface and the obstacles in the vehicle's path, and to process each point cloud in real time. The proposed pipeline aims to solve the problem of 3D point cloud segmentation for data received from a LiDAR in a fast and low complexity manner that targets real world applications. The two-step algorithm first extracts the ground surface in an iterative fashion using deterministically assigned seed points, and then clusters the remaining non-ground points taking advantage of the structure of the LiDAR point cloud. Our proposed algorithms outperform similar approaches in running time, while producing similar results and support the validity of this pipeline as a segmentation tool for real world applications.

In the industrial human-robot collaboration environment, workers and robots often need to collaborate to complete complex tasks in a shared space. Accurate and robust motion recognition is the key to improving the productivity and safety of human-robot collaboration. Accurately tracking and recognizing workers' motions can provide clues about tasks to be performed, thus ensuring the safety of collaborative workspaces. Aiming at the differences in postures and characteristics of workers' motions in the working environment, the human body's self-occlusion and partial occlusion, this paper uses a skeleton recognition algorithm to estimate the 3D skeleton information in monocular video. A novel skeleton descriptor that describes static and dynamic features of the 3D skeletons in the short time around the current frame is proposed. Besides, we train a deep neural network to recognize the human motion. Experimental results show that the proposed skeleton descriptor can improve the accuracy of motion recognition by about 4%, and the neural network we use outperforms the other three classical classification methods.

Trust has been identified as a key element for the successful cooperation between humans and robots. However, little research has been directed at understanding trust development in industrial human-robot collaboration (HRC). With industrial robots becoming increasingly integrated into production lines as a means for enhancing productivity and quality, it will not be long before close proximity industrial HRC becomes a viable concept. Since trust is a multidimensional construct and heavily dependent on the context, it is vital to understand how trust develops when shop floor workers interact with industrial robots. To this end, in this study a trust measurement scale suitable for industrial HRC was developed in two phases. In phase one, an exploratory study was conducted to collect participants’ opinions qualitatively. This led to the identification of trust related themes relevant to the industrial context and a related pool of questionnaire items was generated. In the second phase, three human-robot trials were carried out in which the questionnaire items were applied to participants using three different types of industrial robots. The results were statistically analysed to identify the key factors impacting trust and from these generate a trust measurement scale for industrial HRC.

Front View Gait Recognition using Spherical Space Model with Human Point Clouds

Autonomous UAVs operating in GPS-denied environments rely heavily on onboard perception for safe navigation, yet traditional LiDAR and camera-based systems falter in poor lighting and weather situations. In contrast, millimetre wave (mmWave) radar offers superior robustness and low power consumption but produces extremely sparse, noisy, and low-resolution measurements that limit its utility for dense 3D understanding. This work proposes a novel radar-only inference framework that reconstructs dense LiDAR-like point clouds by leveraging a stereo millimeter-wave radar (IWR6843AoP) data with hallucinated Time-of-Flight (ToF) representations. Rather than relying on explicit ToF sensors, our method learns to infer mid-range depth priors through an auxiliary supervised hallucination network trained with radar inputs only. A unified PointNet-based architecture fuses stereo radar features with the hallucinated ToF embedding to generate dense Cartesian point clouds in real time. A large-scale, high-fidelity multimodal dataset of approximately 50,000 frames is generated in NVIDIA Isaac Sim to emulate realistic UAV sensor configurations. Experimental evaluations demonstrate strong robustness to radar noise and reliable geometric consistency suitable for downstream tasks such as ICP-based odometry. The resulting framework delivers LiDAR-like geometric structure using only radar at runtime, offering a computationally efficient, deployable, and resilient solution for UAV navigation in GPS-denied and harsh environmental conditions.

No abstract available.

Simulation Platform to Investigate Safe Operation of Human-Robot Collaboration Systems

Light detection and ranging (LiDAR) sensors provide accurate 3-D point clouds for noncooperative spacecraft pose estimation. Several robust methods, such as iterative closest point, exist to perform a local refinement of the pose starting from an initial estimate. However, finding the initial pose of the spacecraft is a global optimization problem, which is challenging to solve in real time. This is especially true on space hardware with limited computing power. In addition, many spacecrafts have a shape with multiple symmetries, making an unambiguous initial pose estimation impossible. This work introduces a convolutional-neural-network-based pose estimation method, accounting for potential symmetries of the target satellite. The point clouds are projected to a 2-D depth image before being processed by the network. To generate a sufficient amount of training data, a LiDAR simulator integrating multiple effects such as reflections or laser beam divergence is developed. While being trained solely on synthetic point clouds, the pose estimation method shows to be precise, efficient, and reliable when evaluated on real point clouds taken at a hardware-in-the-loop rendezvous test facility. A runtime evaluation on potential space computing hardware is also performed to demonstrate the applicability of the method to real-time onboard pose estimation.

Depth cameras started to gain popularity in agricultural applications during the last years. This type of cameras has been implemented mainly for three-dimensional (3D) reconstruction of objects in indoor or outdoor sceneries. The use of such cameras in the construction of 3D models for simulation purposes of complex structures that are usually met in nature, such as trees and other plants, is a great challenge. Remarkably, agricultural environments are extremely complex. Thus, the proper setup and implementation of such technologies is particularly important in order to attain useable data. So far, the depth information collected using these cameras varies among different objects’ structure and sensing conditions due to the uncertainty of the outdoor environment. The use of a specific methodology using color and depth images gives the opportunity to extract geometrical characteristics information about point clouds of the targeted objects. This chapter explores the different technologies used by depth cameras and presents several applications concerning indoor and outdoor environments by presenting indicative scenarios for agricultural applications. Towards that direction, a 3D reconstruction of trees was established producing point clouds from Red Green Blue Depth (RGB-D) images acquired in real field conditions. The point cloud samples of trees were collected using an unmanned ground vehicle (UGV) and imported in Gazebo in order to visualize a simulation of the environment. This simulation technique can be used for testing and evaluating the navigation of robotic systems. By further analyzing the resulted 3D point clouds, various geometrical measurements of the simulated samples, such as the volume or the height of tree canopies, can be calculated. Possible weaknesses of this procedure are mainly attributed to the camera’s limitations and the sampling parameters. However, results show that it is possible to establish a suitable simulation environment to implement it in several agricultural applications by utilizing automated unmanned robotic platforms.KeywordsDepth camerasUnmanned ground vehiclesScene reconstruction3D point cloudAgricultural robotsSituation awareness

Abstract. The Project ADFEX (Adaptive Federative 3D Exploration of Multi Robot System) pursues the goal to develop a time- and cost-efficient system for exploration and monitoring task of unknown areas or buildings. A fleet of unmanned aerial vehicles equipped with appropriate sensors (laser scanner, RGB camera, near infrared camera, thermal camera) were designed and built. A typical operational scenario may include the exploration of the object or area of investigation by an UAV equipped with a laser scanning range finder to generate a rough point cloud in real time to provide an overview of the object on a ground station as well as an obstacle map. The data about the object enables the path planning for the robot fleet. Subsequently, the object will be captured by a RGB camera mounted on the second flying robot for the generation of a dense and accurate 3D point cloud by using of structure from motion techniques. In addition, the detailed image data serves as basis for a visual damage detection on the investigated building. This paper focuses on our experience with use of a low-cost light-weight Hokuyo laser scanner onboard an UAV. The hardware components for laser scanner based 3D point cloud acquisition are discussed, problems are demonstrated and analyzed, and a quantitative analysis of the accuracy potential is shown as well as in comparison with structure from motion-tools presented.

Automatic recognition of machining features is the key technology to realize the integration of CAD/CAM. With the development of intelligent manufacturing technology, it has important significance to automatically recognize machining features from parts represented by real-time point clouds. At the same time, it puts new challenges for feature recognition technology. In this paper, based on the 3D point cloud data of part models and PointNet architecture, an approach and data structure for automatic recognition of machining features using convolution neural networks (CNN) is proposed. A sample library for learning 3D point cloud data is constructed by CAD model transformation and feature sampling. The presented CNN recognition system can recognize twenty-four kinds of machined features by sample training and recognition experiments. The recognition accuracy rate is more than 95%. This approach makes full use of the invariance of transformation of the point cloud model, and reduces the unnecessary data that is used by conventional 3D voxel network models. The presented method is simple and efficient, and has good robustness to the point cloud disturbance and noise.

This paper introduces at first GQS (General Quadric Surface) equation, then considers that utilizing depth images referenced to various datum planes to model point cloud. After getting reference datum, this paper proposing the methods of creating depth images from point cloud and the specific steps with reference to different datums. Finally we choose some point cloud data of ancient building components from laser scanning data of Forbidden City in China as experiment data, and develop a depth images modeling system. Experiment results demonstrate the stability and high precision of the method of plane, cylinder and sphere fitting as well as the validity of depth images to represent point cloud of object.