FAst Detection Algorithm for 3D Keypoints (FADA-3K)

The state-of-art 6D object pose detection methods use convolutional neural networks to estimate objects' 6D poses from RGB images. However, they require huge numbers of images with explicit 3D annotations such as 6D poses, 3D bounding boxes and 3D keypoints, either obtained by manual labeling or inferred from synthetic images generated by 3D CAD models. Manual labeling for a large number of images is a laborious task, and we usually do not have the corresponding 3D CAD models of objects in real environment. In this paper, we develop a keypoint-based 6D object pose detection method (and its deep network) called Object Keypoint based POSe Estimation (OK-POSE). OK-POSE employs relative transformation between viewpoints for training. Specifically, we use pairs of images with object annotation and relative transformation information between their viewpoints to automatically discover objects' 3D keypoints which are geometrically and visually consistent. Then, the 6D object pose can be estimated using a keypoint-based geometric reasoning method with a reference viewpoint. The relative transformation information can be easily obtained from any cheap binocular cameras or most smartphone devices, thus greatly lowering the labeling cost. Experiments have demonstrated that OK-POSE achieves acceptable performance compared to methods relying on the object's 3D CAD model or a great deal of 3D labeling. These results show that our method can be used as a suitable alternative when there are no 3D CAD models or a large number of 3D annotations.

In the presented work, innovative low-sampling lightweight architecture is proposed to solve the task of 3D human pose estimation based on 2D key points. The approach introduces specialized trainable pose encodings designed for 3D pose estimation tasks, which are used in conjunction with traditional pose encodings to represent the input data. The architecture of the method includes multilevel feature processing and their adaptive association using a spatial attention mechanism, which allows to enhance relevant features. Experiments conducted on standard test datasets confirmed the effectiveness of the proposed method: a mean joint position error (MPJPE) value of 42.1 was achieved, which exceeds the results of existing approaches.

Object 6D pose estimation is an important research topic in the field of computer vision due to its wide application requirements and the challenges brought by complexity and changes in the real-world. We think fully exploring the characteristics of spatial relationship between points will help to improve the pose estimation performance, especially in the scenes of background clutter and partial occlusion. But this information was usually ignored in previous work using RGB image or RGB-D data. In this paper, we propose a framework for 6D pose estimation from RGB-D data based on spatial structure characteristics of 3D keypoints. We adopt point-wise dense feature embedding to vote for 3D keypoints, which makes full use of the structure information of the rigid body. After the direction vectors pointing to the keypoints are predicted by CNN, we use RANSAC voting to calculate the coordinate of the 3D keypoints, then the pose transformation can be easily obtained by the least square method. In addition, a spatial dimension sampling strategy for points is employed, which makes the method achieve excellent performance on small training sets. The proposed method is verified on two benchmark datasets, LINEMOD and OCCLUSION LINEMOD. The experimental results show that our method outperforms the state-of-the-art approaches, achieves ADD(-S) accuracy of 98.7% on LINEMOD dataset and 52.6% on OCCLUSION LINEMOD dataset in real-time.

The next generation of service robots is to offer services relying heavily on visually guided manipulation, besides navigation, such services as errand, logistics, appliance, home keeping, etc. For the successful introduction of these services, it is critical to establish the consumer-level dependability in 3D object recognition and pose estimation in a natural setting where a large variation of environment, e.g., perspective, texture, form factor, illumination, occlusion, etc., is common. To address this problem, we propose an approach of the two-layered particle filter to the dependability in 3D object recognition and pose estimation. In the upper layer, a set of object pose candidates is identified and maintained in the search space as a set of super-particles, each of which is assigned a probability of the true pose and evolved in time along with the accumulation of further evidences. To define the object pose candidates, first, we acquire initially weak evidences quickly and interpret them in terms of possible object poses in space. These interpretations serve as the region of interest for detailed investigation by which the pose probabilities are computed for individual interpretations based on the likelihood and unlikelihood of various features available in the corresponding regions of interest. During the process of probability computation, we select the object pose candidates to be used as super-particles in the upper layer. In the lower level, the pose uncertainties associated with the individual candidates are represented as particles that are subject to the propagation in time. Finally, the experimental results support the strength of the proposed approach in the real environment in terms of its dependability in 3D object recognition and pose estimation.

Single-camera object detection makes three-dimensional information into two-dimensional images, which misses the three-dimensional structure information of object. In addition, due to the changes in perspective, scale and attitude, it is difficult to track object robustly. In this paper, we put forward a stereo vison tracking algorithm based on 3-dimensional (3D) Delaunay triangulation. Firstly, 3D key points with Scale Invariant Feature Transform (SIFT) descriptors are extracted in 3D point clouds. Then, 3D Delaunay triangulation is built to represent the spatial configuration information for the feature points. Based on the unique topological structure of 3D Delaunay triangulation of a set of key points, the object representation is obtained. Finally in particle filter framework, 3D target tracking is in process. Here, we fully exploit spatial geometry to restrict the feature points, which build a discriminative model for object tracking. Experiments show that the method has good tolerance for changes in illumination, scale and attitude.

This paper presents an appearance-based method for object pose recognition using single camera image. The basic idea of our method is to find the correlation between object image and its pose, and use it for object recognition and pose estimation. Canonical correlation analysis is introduced to derive such correlation and build a compact appearance model. In the approach, we first obtain a pair of training data set, i.e., object images and their pose parameters. The appearance model is given as the subspace spanned by the canonical vectors that maximize the correlation between images and poses. Pose parameters of currently observed image is predicted by finding the regression coefficient in this subspace. We also introduce the kernel methods to cope with the non-linearity lies in training data set. Experiments are conducted on object pose estimation and vehicle type classification problem. Performance of our appearance models is discussed through the comparison with conventional subspace method

With the advancement of intelligent manufacturing, robots are increasingly involved in industrial production, for example, replacing human beings to identify and manipulate objects. However, it is still a challenging task for object recognition and pose estimation in industrial environment, especially under large environmental noise, mutual occlusions and high-speed requirements. In this paper, we propose a new method for 6D object pose estimation, Super Key 4-Points Congruent Sets (SK-4PCS), which is a fast global method combined with local features. The proposed method includes three steps. Firstly, the key points in 3D point clouds are extracted by local features, which are invariant with respect to rotation and translation. Secondly, the smart indexing scheme is used to realize the query of 4-points congruent sets globally. Finally, the coarse pose obtained quickly is further refined by the Iteration Closest Point algorithm (ICP) [1]. The proposed method is also compared with existing methods by the evaluation indicator Visible Surface Discrepancy (VSD) on the LINEMOD dataset [2]. The experimental results show that the refined SK-4PCS results in significant speedup and satisfactory accuracy in 6D object pose estimation.

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

Purpose Accurate 6D object pose estimation is essential for various robotic tasks. Uncertain pose estimates can lead to task failures; however, a certain degree of error in the pose estimates is often acceptable. This paper aims to enable the robots to make informed decisions by quantifying errors in the object pose estimate and acceptable errors for task success. Design/methodology/approach In this paper, the authors introduce a framework for evaluating robotic task success under object pose uncertainty, representing both the estimated error space of the object pose and the acceptable error space for task success using multi-modal non-parametric probability distributions. The proposed framework pre-computes the acceptable error space for task success using dynamic simulations and subsequently integrates the pre-computed acceptable error space over the estimated error space of the object pose to predict the likelihood of the task succes. Findings The authors evaluated the proposed framework on two mobile manipulation tasks. Their results show that by representing the estimated and the acceptable error space using multi-modal non-parametric distributions, the authors achieve higher task success rates and fewer failures. Research limitations/implications Their proposed framework is generic and can be applied to a wide range of robotic tasks requiring object pose estimation. Hence, given the recent advancements in object pose uncertainty estimation and dynamic simulations, the proposed framework, in conjunction with these advancements, has the potential to enable robots to make reliable and informed decisions under pose uncertainty. Originality/value Unlike related works that model both acceptable error space and estimated error space using parametric uni-modal distributions, the authors model them as multi-modal distributions which is often the case in the real world.

Object recognition and pose estimation are critical components in autonomous robot manipulation systems, playing a crucial role in enabling robots to interact effectively with the environment. During actual execution, the robot must recognize the object in the current scene, estimate its pose, and then select a feasible grasp pose from the pre-defined grasp configurations. While most existing methods primarily focus on pose estimation, they often neglect the graspability and reachability aspects. This oversight can lead to inefficiencies and failures during execution. In this study, we introduce an innovative graspability-aware object pose estimation framework. Our proposed approach not only estimates the poses of multiple objects in clustered scenes but also identifies graspable areas. This enables the system to concentrate its efforts on specific points or regions of an object that are suitable for grasping. It leverages both depth and color images to extract geometric and appearance features. To effectively combine these diverse features, we have developed an adaptive fusion module. In addition, the fused features are further enhanced through a graspability-aware feature enhancement module. The key innovation of our method lies in improving the discriminability and robustness of the features used for object pose estimation. We have achieved state-of-the-art results on public datasets when compared to several baseline methods. In real robot experiments conducted on a Franka Emika robot arm equipped with an Intel Realsense camera and a two-finger gripper, we consistently achieved high success rates, even in cluttered scenes.

Monocular 3D human pose estimation is challenging due to depth ambiguity. Convolution-based and Graph-Convolution-based methods have been developed to extract 3D information from temporal cues in motion videos. Typically, in the lifting-based methods, most recent works adopt the transformer to model the temporal relationship of 2D keypoint sequences. These previous works usually consider all the joints of a skeleton as a whole and then calculate the temporal attention based on the overall characteristics of the skeleton. Nevertheless, the human skeleton exhibits obvious part-wise inconsistency of motion patterns. It is therefore more appropriate to consider each part's temporal behaviors separately. To deal with such part-wise motion inconsistency, we propose the Part Aware Temporal Attention module to extract the temporal dependency of each part separately. Moreover, the conventional attention mechanism in 3D pose estimation usually calculates attention within a short time interval. This indicates that only the correlation within the temporal context is considered. Whereas, we find that the part-wise structure of the human skeleton is repeating across different periods, actions, and even subjects. Therefore, the part-wise correlation at a distance can be utilized to further boost 3D pose estimation. We thus propose the Part Aware Dictionary Attention module to calculate the attention for the part-wise features of input in a dictionary, which contains multiple 3D skeletons sampled from the training set. Extensive experimental results show that our proposed part aware attention mechanism helps a transformer-based model to achieve state-of-the-art 3D pose estimation performance on two widely used public datasets. The codes and the trained models are released at https://github.com/thuxyz19/3D-HPE-PAA.

Experts predict that future robot applications will require safe and predictable operation: robots will need to be able to explain what they are doing to be trusted. To reach this goal, they will need to perceive their environment and its object to better understand the world and the tasks they have to perform. This article gives an overview of present advances with the focus on options to learn, detect, and grasp objects. With the approach of colour and depth (RGB-D) cameras and the advances in AI and deep learning methods, robot vision has been pushed considerably over the last years. We summarise recent results for pose estimation of objects and work on verifying object poses using a digital twin and physics simulation. The idea is that any hypothesis from an object detector and pose estimator is verified leveraging on the continuous advances in deep learning approaches to create object hypotheses. We then show that the object poses are robust enough such that a mobile manipulator can approach the object and grasp it. We intend to indicate that it is now feasible to model, recognise and grasp many objects with good performance, though further work is needed for applications in industrial settings.

Generalization is very important for pose estimation, especially for 3D pose estimation where small changes in the 2D images could trigger structural changes in the 3D space. To achieve generalization, the system needs to have the capability of detecting estimation errors by double-checking the projection coherence between the 3D and 2D spaces and adapting its network inference process based on this feedback. Current pose estimation is one-time feed-forward and lacks the capability to gather feedback and adapt the inference outcome. To address this problem, we propose to explore the concept of progressive inference where the network learns an observer to continuously detect the prediction error based on constraints matching, as well as an adjuster to refine its inference outcome based on these constraints errors. Within the context of 3D hand pose estimation, we find that this observer-adjuster design is relatively unstable since the observer is operating in the 2D image domain while the adjuster is operating in the 3D domain. To address this issue, we propose to construct two sets of observers-adjusters with complementary constraints from different perspectives. They operate in a dynamic sequential manner controlled by a decision network to progressively improve the 3D pose estimation. We refer to this method as Cross-Constrained Progressive Inference (CCPI). Our extensive experimental results on FreiHAND and HO-3D benchmark datasets demonstrate that the proposed CCPI method is able to significantly improve the generalization capability and performance of 3D hand pose estimation.

In recent times, object detection and pose estimation have gained significant attention in the context of robotic vision applications. Both the identification of objects of interest as well as the estimation of their pose remain important capabilities in order for robots to provide effective assistance for numerous robotic applications ranging from household tasks to industrial manipulation. This problem is particularly challenging because of the heterogeneity of objects having different and potentially complex shapes, and the difficulties arising due to background clutter and partial occlusions between objects. As the main contribution of this work, we propose a system that performs real-time object detection and pose estimation, for the purpose of dynamic robot grasping. The robot has been pre-trained to perform a small set of canonical grasps from a few fixed poses for each object. When presented with an unknown object in an arbitrary pose, the proposed approach allows the robot to detect the object identity and its actual pose, and then adapt a canonical grasp in order to be used with the new pose. For training, the system defines a canonical grasp by capturing the relative pose of an object with respect to the gripper attached to the robot’s wrist. During testing, once a new pose is detected, a canonical grasp for the object is identified and then dynamically adapted by adjusting the robot arm’s joint angles, so that the gripper can grasp the object in its new pose. We conducted experiments using a humanoid PR2 robot and showed that the proposed framework can detect well-textured objects, and provide accurate pose estimation in the presence of tolerable amounts of out-of-plane rotation. The performance is also illustrated by the robot successfully grasping objects from a wide range of arbitrary poses.