Sampling of 3DOF Robot Manipulator Joint-Limits for Haptic Feedback

Object recognition and grasping are important means of interaction between robot and environment, and also two of the main tasks of robot. Due to the rich information provided by depth sensors, it has paved the way for the object recognition. The geometric information is more conducive to the primitive recognition of the object, and the primitive shape information is used as the input information for the robot to grasp. This study proposes a primitive shape recognition method using local point cloud. First, 900 sets of point cloud data including three primitive shapes was created. Then the PointNet network using the point cloud data to recognize the primitive shape of the objects was trained. Experiments in simulation and physical world shows our recognition method can effectively recognize the primitive shape of the object.KeywordsPrimitive shapeObject recognitionPoint cloudPointNet

Research on 3D surface reconstruction and body size measurement of pigs based on multi-view RGB-D cameras

Body size parameters of beef cattle are crucial for assessing growth status and breeding value. In actual farming environments, the various postures of beef cattle and complex backgrounds can affect the accuracy and stability of non-contact body measurement methods. Therefore, this paper proposes a novel method called the cattle body measurement method (CBMM), which combines keypoint detection with local point cloud clustering. First, a keypoint detection model based on YOLOv8-SimBiFPN is constructed. This model enhances the feature extraction and fusion capabilities of YOLOv8-pose by introducing SimAM and BiFPN into the backbone and neck networks, respectively, and realizes 2D keypoint detection for beef cattle in various postures. Second, a 3D keypoint-locating algorithm based on Density-based spatial clustering of applications with noise (DBSCAN) is proposed. This algorithm utilizes 2D keypoints, depth maps and camera parameters to generate local point clouds, which are then clustered using DBSCAN to segment cattle body point clouds, thereby relocating the 3D keypoints based on their positional features. Finally, body size parameters are calculated based on the 3D keypoints and distance formulae. In our experiment, the mean average precision (mAP@0.5) of YOLOv8-SimBiFPN reached 99.1% on an Angus beef cattle keypoint detection dataset. The mean absolute percentage errors for measuring beef cattle withers height, hip height, body depth, body length, and oblique body length using the CBMM were 4.37%, 4.96%, 6.47%, 4.84%, and 4.14%, respectively. In summary, our method can achieve non-contact body measurement for beef cattle in a free-moving state with high accuracy and stability.

Localization is considered as a key capability for autonomous vehicles act in urban environments. Though have been proved to be able to perform convictive results, localization methods using neither laser scanners nor vision sensors could achieve the goal about balancing between accuracy and cost. In this paper, an occupancy grid based localization framework is presented in order to obtain a precise positioning result with relatively low-cost sensor configuration in large scale urban environment. The proposed approach takes a prebuilt global grid map as prior knowledge for localization. Model based feature extraction method is introduced to provide laser points classification, with each extracted point allocated a specified weight to describe local characteristic. The prior grid map is generated from weighted point cloud to be able to describe the local metric features such as curbs and building facades. Localization function is then carried out with a weight point based maximum likelihood matching method to determine the correspondence between local point cloud and the reference grid map. There are also position initialization and reference map management modules to make the framework more practical and reliable. In the end, the proposed approach has been validated by promising experimental results with long distance tests in large urban environments.

The robustness of correspondence-based point cloud registration relies on transformation invariance and intrinsic distinctiveness of the descriptors computed for registration. However, for challenging scenarios with different objects having similar local geometry and low point cloud overlap, existing descriptors struggle to retain these properties. This inevitably leads to a considerable loss in matching inlier rates and compromised registration. To address the issue, we propose RID-Net that computes robust Rotation-Invariant and Distinctive descriptors for point cloud registration. Our model works on the philosophy of locally accurate feature abstraction for the points while also accounting for the surroundings. Specifically, the locally accurate representation generated by our rotation-invariant MLP-based module is further enhanced through a proposed spherical shell attention mechanism to explicitly focus on the surrounding geometry. The overall network is designed by staking the proposed component, while eventually leveraging coarse-to-fine correspondences and a robust pose estimator to compute the ultimate output. With extensive evaluation on indoor, outdoor and synthetic point cloud data, we establish the efficacy of the proposed RID-Net. In particular, our method improves the inlier ratio by about 4% on 3DMatch/3DLoMatch dataset and reduces the rotation and translation errors by 26.1% and 19.7% on the KITTI dataset.

A common scenario in Search and Rescue robotics is to map and patrol a disaster site to assess the situation and plan potential missions of rescue teams. Particular importance has to be given to changes in the environment as these may correspond to critical events like building collapses, movement of objects, etc. This paper presents a change detection pipeline for LiDAR-equipped robots to assist humans in detecting those changes. The local 3D point cloud data is compared to an octree-based occupancy map representation of the environment by computing the Mahalanobis distance to the closest voxel in the map. The thresholded distance is processed by a clustering algorithm to obtain a set of change candidates. Finally, outliers in these sets are filtered using a random forest classifier. Changes are continuously mapped during a sortie based on their classification score and number of occurrences. Changes are reported in real time during robot operation.

At present, 3D vision technology has been more and more applied in the field of livestock breading industry. Three depth cameras are installed in the data acquisition walkway from different views, as the animals pass through the walkway, the depth cameras will get the local point cloud of livestock at the same time. By this method the livestock point cloud exists inevitable noise, as well as the background point cloud including railing and ground. Therefore, the segmentation of livestock point cloud from the original point cloud mixed with noise and background is the important first step of the application of three-dimensional vision in livestock breading industry. This paper proposed using the PointNet++ model to separate the target point cloud from the background point cloud. The improved DBSCAN clustering algorithm is proposed to denoise and segment the data preliminarily, which is helpful to label the data manually. The labeled data are used to train the PointNet++ neural network model to extract target point cloud efficiently and accurately. The experimental results show that the mean Intersection over Union (MIOU) of pig target point cloud segmentation based on single-scale grouping and multi-scale grouping Pointnet++ are 0.932 and 0.940, respectively, which both obtain good segmentation results.

It is speculated that virtual environments (VE) might be used as a training tool in brain injury rehabilitation. The rehabilitation process often involves practicing so-called instrumental activities of daily living (IADL), such as shopping, cooking, and using a telephone. If a brain injury patient is to practice such activities in a VE, the patient must be able to navigate the viewpoint and interact with virtual objects in an understandable way. People with brain injury may be less tolerant to a poor interface and a VE might therefore become unusable due to, for example, an unsuitable input device. In this paper we present two studies aimed to do initial usability testing of VE interaction methods on people without experience of 3D computer graphics. In the first study four navigation input device configurations were compared: the IntelliKeys keyboard and the Microsoft Sidewinder joystick, both programmed with two and three degrees of freedom (DOF). The purpose of the second study was to evaluate a method for interaction with objects, and to find a sufficiently usable input device for this purpose. The keyboard was found to be more suitable for navigation tasks in which the user wants to give the viewpoint a more advantageous position and orientation for carrying out a specific task. No big differences could be found between two and three DOFs. The method for interaction with objects was found to work sufficiently well. No difference in performance could be found between mouse and touch screen, but some evidence was found that they affect the usability of the VE interface in different ways.

The objective of this research project is to develop a user-friendly and cost-effective interactive input device that allows intuitive and efficient manipulation of 3D objects (6 DoF) in virtual reality (VR) visualization environments with flat projections walls. During this project, it was planned to develop an extended version of a laser pointer with multiple laser beams arranged in specific patterns. Using stationary cameras observing projections of these patterns from behind the screens, it is planned to develop an algorithm for reconstruction of the emitter’s absolute position and orientation in space. Laser pointer concept is an intuitive way of interaction that would provide user with a familiar, mobile and efficient navigation though a 3D environment. In order to navigate in a 3D world, it is required to know the absolute position (x, y and z position) and orientation (roll, pitch and yaw angles) of the device, a total of 6 degrees of freedom (DoF). Ordinary laser-based pointers when captured on a flat surface with a video camera system and then processed, will only provide x and y coordinates effectively reducing available input to 2 DoF only. In order to overcome this problem, an additional set of multiple (invisible) laser pointers should be used in the pointing device. These laser pointers should be arranged in a way that the projection of their rays will form one fixed dot pattern when intersected with the flat surface of projection screens. Images of such a pattern will be captured via a real-time camera-based system and then processed using mathematical re-projection algorithms. This would allow the reconstruction of the full absolute 3D pose (6 DoF) of the input device. Additionally, multi-user or collaborative work should be supported by the system, would allow several users to interact with a virtual environment at the same time. Possibilities to port processing algorithms into embedded processors or FPGAs will be investigated during this project as well.

PurposeThis paper aims to present an iterative path-following method with joint limits to solve the problem of large computation cost, movement exceeding joint limits and poor path-following accuracy for the path planning of hyper-redundant snake-like manipulator.Design/methodology/approachWhen a desired path is given, new configuration of the snake-like manipulator is obtained through a geometrical approach, then the joints are repositioned through iterations until all the rotation angles satisfy the imposed joint limits. Finally, a new arrangement is obtained through the analytic solution of the inverse kinematics of hyper-redundant manipulator. Finally, simulations and experiments are carried out to analyze the performance of the proposed path-following method.FindingsSimulation results show that the average computation time is 0.1 ms per step for a hyper-redundant manipulator with 12 degrees of freedom, and the deviation in tip position can be kept below 0.02 mm. Experiments show that all the rotation angles are within joint limits.Research limitations/implicationsCurrently , the manipulator is working in open-loop, the elasticity of the driving cable will cause positioning error. In future, close-loop control based on real-time attitude detection will be used in in combination with the path-following method to achieve high-precision trajectory tracking.Originality/valueThrough a series of iterative processes, the proposed method can make the manipulator approach the desired path as much as possible within the joint constraints with high precision and less computation time.

The development of Internet of Things (IoT) technology has enabled intelligent robots to have more sensing and decision-making capabilities, broadening the application areas of robots. Grasping operation is one of the basic tasks of intelligent robots, and vision-based robot grasping technology can enable robots to perform dexterous grasping. Compared with 2D images, 3D point clouds based on objects can generate more reasonable and stable grasping poses. In this paper, we propose a new algorithm structure based on the PointNet network to process object point cloud information. First, we use the T-Net network to align the point cloud to ensure its rotation invariance; then we use a multilayer perceptron to extract point cloud characteristics and use the symmetric function to get global features, while adding the point cloud characteristics attention mechanism to make the network more focused on the object local point cloud. Finally, a grasp quality evaluation network is proposed to evaluate the quality of the generated candidate grasp positions, and the grasp with the highest score is obtained. A grasping dataset is generated based on the YCB dataset to train the proposed network, which achieves excellent classification accuracy. The actual grasping experiments are carried out using the Baxter robot and compared with the existing methods; the proposed method achieves good grasping effect.

Recovering an outdoor environment’s surface mesh is vital for an agricultural robot during task planning and remote visualization. Image-based dense 3D reconstruction is sensitive to large movements between adjacent frames and the quality of the estimated depth maps. Our proposed solution for these problems is based on a newly-designed panoramic stereo camera along with a hybrid novel software framework that consists of three fusion modules: disparity fusion, pose fusion, and volumetric fusion. The panoramic stereo camera with a pentagon shape consists of 5 stereo vision camera pairs to stream synchronized panoramic stereo images for the following three fusion modules. In the disparity fusion module, rectified stereo images produce the initial disparity maps using multiple stereo vision algorithms. Then, these initial disparity maps, along with the intensity images, are input into a disparity fusion network to produce refined disparity maps. Next, the refined disparity maps are converted into full-view (360°) point clouds or single-view (72°) point clouds for the pose fusion module. The pose fusion module adopts a two-stage global-coarse-to-local-fine strategy. In the first stage, each pair of full-view point clouds is registered by a global point cloud matching algorithm to estimate the transformation for a global pose graph’s edge, which effectively implements loop closure. In the second stage, a local point cloud matching algorithm is used to match single-view point clouds in different nodes. Next, we locally refine the poses of all corresponding edges in the global pose graph using three proposed rules, thus constructing a refined pose graph. The refined pose graph is optimized to produce a global pose trajectory for volumetric fusion. In the volumetric fusion module, the global poses of all the nodes are used to integrate the single-view point clouds into the volume to produce the mesh of the whole garden. The proposed framework and its three fusion modules are tested on a real outdoor garden dataset to show the superiority of the performance. The whole pipeline takes about 4 min on a desktop computer to process the real garden dataset, which is available at: https://github.com/Canpu999/Trimbot-Wageningen-SLAM-Dataset.

Smart mobile device and consumer level 3D visual sensor are becoming very cheap and widely accessible in people's daily lives. Visual information based indoor localization is getting more and more attention in the computer vision research community to achieve real time and precise localization result. In this paper, we present a 6-degree-of-freedom (6-DoF) pose estimation using a portable 3D visual sensor mounted on a mobile device. A detailed 3D model and a WiFi received signal strength model of the indoor environment are constructed in the offline training phase. During the online localization, we first use WiFi signals to locate the device in a 3D sub model. The initial pose is calculated through feature matching between the online captured 2D image and the key frame images used to build the 3D model. Then we employ iterative closest point (ICP) algorithm to estimate the rigid transform between online captured 3D point cloud and local 3D point cloud. The 6-DOF pose estimation is further refined by random sample consensus (RANSAC) algorithm. We implement the system on iOS platform and the experiments carried out in an indoor environment shows promising result of our approach. To the best of our knowledge, this is the first portable 3D visual sensor based indoor localization system on mobile device.

The goal of human-robot motion retargeting is to let a robot follow the movements performed by a human subject. This is traditionally achieved by applying the estimated poses from a human pose tracking system to a robot via explicit joint mapping strategies. In this paper, we present a novel approach that combine the human pose estimation and the motion retarget procedure in a unified generative framework. A 3D parametric human-robot model is proposed that has the specific joint and stability configurations as a robot while its shape resembles a human subject. Using a single depth camera to monitor human pose, we use its raw depth map as input and drive the human-robot model to fit the input 3D point cloud. The calculated joint angles of the fitted model can be applied onto the robots for retargeting. The robot's joint angles, instead of fitted individually, are fitted globally so that the transformed surface shape is as consistent as possible to the input point cloud. The robot configurations including its skeleton proportion, joint limitation, and DoF are enforced implicitly in the formulation. No explicit and pre-defined joints mapping strategies are needed. This framework is tested with both simulations and real robots that have different skeleton proportion and DoFs compared with human to show its effectiveness for motion retargeting.

Surgical simulators for minimally invasive surgery have been developing in the 1990s. Most of them use high-end UNIX workstations for real-time simulation of complex human organ models. Only few of them have input devices with force feedback. Recently, personal computer technologies have made real-time display of relatively complex models feasible. We are developing an Intel-based laparoscopic surgical simulator that provides near real-time intuitive interaction between the trainee and simulated models of human organs. The surgical simulator has a prototypical scenario of cholecystectomic surgery. It can interactively simulate the deformation and cutting of cystic duct and vein. In addition, a set of input devices with force feedback has been designed and tested to imitate the manipulation of surgical instruments. The input device has five degrees of freedom and three of them are driven by DC motors to produce force feedback.