Estimation of 6D Object Pose Using a 2D Bounding Box

We propose a single-shot approach for simultaneously detecting an object in an RGB image and predicting its 6D pose without requiring multiple stages or having to examine multiple hypotheses. Unlike a recently proposed single-shot technique for this task [10] that only predicts an approximate 6D pose that must then be refined, ours is accurate enough not to require additional post-processing. As a result, it is much faster - 50 fps on a Titan X (Pascal) GPU - and more suitable for real-time processing. The key component of our method is a new CNN architecture inspired by [27, 28] that directly predicts the 2D image locations of the projected vertices of the object's 3D bounding box. The object's 6D pose is then estimated using a PnP algorithm. For single object and multiple object pose estimation on the LINEMOD and OCCLUSION datasets, our approach substantially outperforms other recent CNN-based approaches [10, 25] when they are all used without postprocessing. During post-processing, a pose refinement step can be used to boost the accuracy of these two methods, but at 10 fps or less, they are much slower than our method.

The next generation of service robots is to offer services relying heavily on visually guided manipulation, besides navigation, such services as errand, logistics, appliance, home keeping, etc. For the successful introduction of these services, it is critical to establish the consumer-level dependability in 3D object recognition and pose estimation in a natural setting where a large variation of environment, e.g., perspective, texture, form factor, illumination, occlusion, etc., is common. To address this problem, we propose an approach of the two-layered particle filter to the dependability in 3D object recognition and pose estimation. In the upper layer, a set of object pose candidates is identified and maintained in the search space as a set of super-particles, each of which is assigned a probability of the true pose and evolved in time along with the accumulation of further evidences. To define the object pose candidates, first, we acquire initially weak evidences quickly and interpret them in terms of possible object poses in space. These interpretations serve as the region of interest for detailed investigation by which the pose probabilities are computed for individual interpretations based on the likelihood and unlikelihood of various features available in the corresponding regions of interest. During the process of probability computation, we select the object pose candidates to be used as super-particles in the upper layer. In the lower level, the pose uncertainties associated with the individual candidates are represented as particles that are subject to the propagation in time. Finally, the experimental results support the strength of the proposed approach in the real environment in terms of its dependability in 3D object recognition and pose estimation.

In order to solve robotic bin-picking problem in many industrial applications, accurate 6D object pose estimation is one of fundamental technologies. This paper presents a method for accurate 6D pose estimation from a single RGB image for texture-less industrial parts. These objects are common but still challenging to deal with, due to the fact that poor surface texture and brightness makes difficult to compute discriminative local appearance descriptors. The proposed method mainly consists of two stages, which ranges from the detection stage to the optimization stage. Firstly, all known objects in the RGB image are detected with 2D bounding box via a tiny convolutional neural network. Then, the second stage will optimize the 6D pose in the Edge image given several coarse initializations. These coarse initializations are generated from the Edge image via a hypothesis-evaluation scheme. Furthermore, the proposed method is validated by achieving state-of-the-art results of texture-less industrial parts for RGB input. According to practical experiments, the proposed method is accurate and robust enough to be applied on the robotic manipulation platform to complete a simple assembly task.

3D human pose estimation aims to reconstruct the human skeleton of all the individuals in a scene by detecting several body joints. The creation of accurate and efficient methods is required for several real-world applications including animation, human–robot interaction, surveillance systems or sports, among many others. However, several obstacles such as occlusions, random camera perspectives, or the scarcity of 3D labelled data, have been hampering the models’ performance and limiting their deployment in real-world scenarios. The higher availability of cameras has led researchers to explore multi-view solutions due to the advantage of being able to exploit different perspectives to reconstruct the pose. Most existing reviews focus mainly on monocular 3D human pose estimation and a comprehensive survey only on multi-view approaches to determine the 3D pose has been missing since 2012. Thus, the goal of this survey is to fill that gap and present an overview of the methodologies related to 3D pose estimation in multi-view settings, understand what were the strategies found to address the various challenges and also, identify their limitations. According to the reviewed articles, it was possible to find that most methods are fully-supervised approaches based on geometric constraints. Nonetheless, most of the methods suffer from 2D pose mismatches, to which the incorporation of temporal consistency and depth information have been suggested to reduce the impact of this limitation, besides working directly with 3D features can completely surpass this problem but at the expense of higher computational complexity. Models with lower supervision levels were identified to overcome some of the issues related to 3D pose, particularly the scarcity of labelled datasets. Therefore, no method is yet capable of solving all the challenges associated with the reconstruction of the 3D pose. Due to the existing trade-off between complexity and performance, the best method depends on the application scenario. Therefore, further research is still required to develop an approach capable of quickly inferring a highly accurate 3D pose with bearable computation cost. To this goal, techniques such as active learning, methods that learn with a low level of supervision, the incorporation of temporal consistency, view selection, estimation of depth information and multi-modal approaches might be interesting strategies to keep in mind when developing a new methodology to solve this task. • First review only on multi-view, multi-modal methods to estimate 3D pose since 2012. • Multi-view allows capturing the full body geometry, making 3D pose estimation easier. • Real-world applications include sports, broadcasting, rehabilitation or animation. • Finding a fast, accurate method with low computational cost remains a challenge. • Multi-modal methods or view selection can lead to an efficient and effective model.

In this paper, we propose a 3D object pose estimation method based on the frustum point cloud for industrial picking robot to grab the express. The core of our idea is to segment the point cloud of the object which should to be pick up first, then extract the best grab plane and grab point on the object point cloud. First of all, a mature 2D detection framework Faster RCNN is used to detect the object in RGB images. Then the 2D bounding box and category of the object closest to the camera will be output. On the basis of this, the frustum point cloud data of the detected object is generated by the combination of the depth map and the projection relationship of the RGB-D camera. Then we used PointNet network to extract the feature of each point in the generated frustum point cloud. Finally, the instance segmentation of the point cloud and the subsequent extraction of the best grab plane and grab point are performed. We show results on both real dataset and synthetic dataset which we collected by ourselves, and our 3D instance segmentation network accuracy on real dataset reached 88%.

Recently, the emergence of Kinect systems has demonstrated the benefits of predicting an intermediate body part labeling for 3D human pose estimation, in conjunction with RGB-D imagery. The availability of depth information plays a critical role, so an important question is whether a similar representation can be developed with sufficient robustness in order to estimate 3D pose from RGB images. This paper provides evidence for a positive answer, by leveraging (a) 2D human body part labeling in images, (b) second-order label-sensitive pooling over dynamically computed regions resulting from a hierarchical decomposition of the body, and (c) iterative structured-output modeling to contextualize the process based on 3D pose estimates. For robustness and generalization, we take advantage of a recent large-scale 3D human motion capture dataset, Human3.6M[18] that also has human body part labeling annotations available with images. We provide extensive experimental studies where alternative intermediate representations are compared and report a substantial 33% error reduction over competitive discriminative baselines that regress 3D human pose against global HOG features.

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

It is widely shown that Convolutional Neural Networks (CNNs) are vulnerable to adversarial examples on most recognition tasks, such as image classification and segmentation. However, few work studies the more complicated task – 3D human pose estimation. This task often requires large-scale datasets, specialized network architectures, and it can be solved either from single-view RGB images or from multi-view RGB images. In this paper, we make the first attempt to investigate the robustness of current state-of-the-art 3D human pose estimation methods. To this end, we build four representative baseline models, where most of the current methods can be generally classified as one of them. Furthermore, we design targeted adversarial attacks to detect whether 3D pose estimators are robust to different camera parameters. For different types of methods, we present a comprehensive study of their robustness on the large-scale Human3.6M benchmark. Our work shows that different methods vary significantly in their resistance to adversarial attacks. Through extensive experiments, we show that multi-view 3D pose estimators can be more vulnerable to adversarial examples. We believe that our efforts can shed light on future works to design more robust 3D human pose estimators.

Weakly-supervised pre-training for 3D human pose estimation via perspective knowledge

We consider the problem of 3D object pose estimation. While much recent work has focused on the RGB domain, the reliance on accurately annotated images limits their generalizability and scalability. On the other hand, the easily available CAD models of objects are rich sources of data, providing a large number of synthetically rendered images. In this paper, we solve this key problem of existing methods requiring expensive 3D pose annotations by proposing a new method that matches RGB images to CAD models for object pose estimation. Our key innovations compared to existing work include removing the need for either real-world textures for CAD models or explicit 3D pose annotations for RGB images. We achieve this through a series of objectives that learn how to select keypoints and enforce viewpoint and modality invariance across RGB images and CAD model renderings. We conduct extensive experiments to demonstrate that the proposed method can reliably estimate object pose in RGB images, as well as generalize to object instances not seen during training.

Object 6D pose estimation is an important research topic in the field of computer vision due to its wide application requirements and the challenges brought by complexity and changes in the real-world. We think fully exploring the characteristics of spatial relationship between points will help to improve the pose estimation performance, especially in the scenes of background clutter and partial occlusion. But this information was usually ignored in previous work using RGB image or RGB-D data. In this paper, we propose a framework for 6D pose estimation from RGB-D data based on spatial structure characteristics of 3D keypoints. We adopt point-wise dense feature embedding to vote for 3D keypoints, which makes full use of the structure information of the rigid body. After the direction vectors pointing to the keypoints are predicted by CNN, we use RANSAC voting to calculate the coordinate of the 3D keypoints, then the pose transformation can be easily obtained by the least square method. In addition, a spatial dimension sampling strategy for points is employed, which makes the method achieve excellent performance on small training sets. The proposed method is verified on two benchmark datasets, LINEMOD and OCCLUSION LINEMOD. The experimental results show that our method outperforms the state-of-the-art approaches, achieves ADD(-S) accuracy of 98.7% on LINEMOD dataset and 52.6% on OCCLUSION LINEMOD dataset in real-time.

Three-dimensional (3D) pose estimation has been widely used in many three-dimensional human motion analysis applications, where inertia-based path estimation is gradually being adopted. Systems based on commercial inertial measurement units (IMUs) usually rely on dense and complex wearable sensors and time-consuming calibration, causing intrusions to the subject and hindering free body movement. The sparse IMUs-based method has drawn research attention recently. Existing sparse IMUs-based three-dimensional pose estimation methods use neural networks to obtain human poses from temporal feature information. However, these methods still suffer from issues, such as body shaking, body tilt, and movement ambiguity. This paper presents an approach to improve three-dimensional human pose estimation by fusing temporal and spatial features. Based on a multistage encoder–decoder network, a temporal convolutional encoder and human kinematics regression decoder were designed. The final three-dimensional pose was predicted from the temporal feature information and human kinematic feature information. Extensive experiments were conducted on two benchmark datasets for three-dimensional human pose estimation. Compared to state-of-the-art methods, the mean per joint position error was decreased by 13.6% and 19.4% on the total capture and DIP-IMU datasets, respectively. The quantitative comparison demonstrates that the proposed temporal information and human kinematic topology can improve pose accuracy.

This paper presents an appearance-based method for object pose recognition using single camera image. The basic idea of our method is to find the correlation between object image and its pose, and use it for object recognition and pose estimation. Canonical correlation analysis is introduced to derive such correlation and build a compact appearance model. In the approach, we first obtain a pair of training data set, i.e., object images and their pose parameters. The appearance model is given as the subspace spanned by the canonical vectors that maximize the correlation between images and poses. Pose parameters of currently observed image is predicted by finding the regression coefficient in this subspace. We also introduce the kernel methods to cope with the non-linearity lies in training data set. Experiments are conducted on object pose estimation and vehicle type classification problem. Performance of our appearance models is discussed through the comparison with conventional subspace method

Deep 3D human pose estimation: A review

With the advancement of intelligent manufacturing, robots are increasingly involved in industrial production, for example, replacing human beings to identify and manipulate objects. However, it is still a challenging task for object recognition and pose estimation in industrial environment, especially under large environmental noise, mutual occlusions and high-speed requirements. In this paper, we propose a new method for 6D object pose estimation, Super Key 4-Points Congruent Sets (SK-4PCS), which is a fast global method combined with local features. The proposed method includes three steps. Firstly, the key points in 3D point clouds are extracted by local features, which are invariant with respect to rotation and translation. Secondly, the smart indexing scheme is used to realize the query of 4-points congruent sets globally. Finally, the coarse pose obtained quickly is further refined by the Iteration Closest Point algorithm (ICP) [1]. The proposed method is also compared with existing methods by the evaluation indicator Visible Surface Discrepancy (VSD) on the LINEMOD dataset [2]. The experimental results show that the refined SK-4PCS results in significant speedup and satisfactory accuracy in 6D object pose estimation.