Skeleton-based Methods for Speaker ActionClassification on Lecture Videos

Comunicació presentada a la 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), celebrada els dies 30 de maig a 3 de juny de 2017 a Washington DC, EUA.

Aiming at the difficulty and low efficiency of pose estimation in the process of automatic drug picking, a pose estimation method combining plane segmentation and key point matching is proposed. First, a point cloud hierarchical clustering algorithm is used to segment the plane area, and the improved secondary key point matching method is used to identify and locate the pill box in each thread, which is convenient for multi-thread processing. Then the point cloud plane normal vector estimation method based on least squares is used to realize the posture reconstruction of the pill box. 600 images with viewpoint change, rotation transformation and scale change were collected for testing, the experimental results show that the pose estimation speed of the medicine box can reach 11 frames/s, the positioning deviation is less than 5 mm, and the pose estimation deviation is within 2°. So the method is efficient and can accurately identify the pose of the medicine box.

Pose detection of objects is an important topic in object-level mapping and indoor localization. In the past, pose estimation methods were performed either with the help of artificial markers or natural features found on the object. However, due to the fact that the markers can only be utilized in controlled environment experiments, the application of marker-based approaches is very limited. Furthermore, methods that depend on the object's natural visual features require texture on the object and lack robustness to illumination and camera viewpoint variations. With the advent of Deep Learning (DL), the classical pose estimation methods have been outperformed. The DL-based pose estimation can detect deep features of the object and exhibits higher robustness to many distortions and variabilities caused by the changes in the illumination and viewpoint conditions. However, the massive training data set requirement is the main challenge with most DL-based methods. The training set is often a real set of images that have been manually labeled or annotated. In addition, such methods face problems related to the degradation of their predicted accuracy in the presence of uncertainties due to the symmetrical structure of many objects. To address the aforementioned issues, a novel and very fast method for generating synthetic data, as well as a contour-based technique for accurate pose estimation (that can handle pose ambiguities for a symmetrical object) are proposed in this paper. The tests that are conducted in multiple indoor scenarios demonstrate not only the effectiveness of the synthetic data generation but also exhibit, in many cases, the very high accuracy of the proposed pose estimation method.

Vehicle pose estimation with light detection and ranging (LiDAR) is essential in the perception technology of autonomous driving. However, because of incomplete observation measurements and sparsity of the LiDAR point cloud, it is challenging to achieve satisfactory pose extraction based on 3D LiDAR with the existing pose estimation methods. In addition, the demand for real-time performance further increases the difficulty of the pose estimation task. In this paper, we propose a novel vehicle pose estimation method based on the convex hull. The extracted 3D cluster is reduced to the convex hull, reducing the subsequent computation burden while preserving essential contour information. Subsequently, a novel criterion based on the minimum occlusion area is developed for the search-based algorithm, enabling accurate pose estimation. Additionally, this criterion renders the proposed algorithm particularly well-suited for obstacle avoidance. The proposed algorithm is validated on the KITTI dataset and a manually labeled dataset acquired at an industrial park. The results demonstrate that our proposed method can achieve better accuracy than the classical pose estimation method while maintaining real-time speed.

We present the first method to perform automatic 3D pose, shape and texture capture of animals from images acquired in-the-wild. In particular, we focus on the problem of capturing 3D information about Grevy's zebras from a collection of images. The Grevy's zebra is one of the most endangered species in Africa, with only a few thousand individuals left. Capturing the shape and pose of these animals can provide biologists and conservationists with information about animal health and behavior. In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other. To overcome these challenges, we integrate the recent SMAL animal model into a network-based regression pipeline, which we train end-to-end on synthetically generated images with pose, shape, and background variation. Going beyond state-of-the-art methods for human shape and pose estimation, our method learns a shape space for zebras during training. Learning such a shape space from images using only a photometric loss is novel, and the approach can be used to learn shape in other settings with limited 3D supervision. Moreover, we couple 3D pose and shape prediction with the task of texture synthesis, obtaining a full texture map of the animal from a single image. We show that the predicted texture map allows a novel per-instance unsupervised optimization over the network features. This method, SMALST (SMAL with learned Shape and Texture) goes beyond previous work, which assumed manual keypoints and/or segmentation, to regress directly from pixels to 3D animal shape, pose and texture. Code and data are available at https://github.com/silviazuffi/smalst.

To improve the accuracy of implant placement in Total Hip Replacement (THR) surgeries, this paper proposes a computer-aided visual system for THR which is composed of a customized acetabular cup, a multi-sensor femoral head trial and a computer for data processing and display. The customized trial is of the same size as the real prosthesis. An image sensor, a gyroscope and an e-compass (including an accelerometer and a magnetometer) are adopted in the femoral head trial. Reference patterns are designed and printed on the internal surface of the cup, whose images are taken by the image sensor for estimation of relative pose and position between the femoral head trial and the acetabulum cup. Two methods of pose estimation are adopted in this system: one based on images and the other based on motion data from gyroscope and e-compass. The efficient perspective-n-point (EPNP) algorithm is used in the image-based pose estimation and achieves a rotation relative error of less than 8% and a translation relative error of less than 10%. The complementary algorithm is adopted in the motion-based pose estimation to smooth the results. Experimental results verified the proposed system.

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Light detection and ranging (LiDAR) sensors provide accurate 3-D point clouds for noncooperative spacecraft pose estimation. Several robust methods, such as iterative closest point, exist to perform a local refinement of the pose starting from an initial estimate. However, finding the initial pose of the spacecraft is a global optimization problem, which is challenging to solve in real time. This is especially true on space hardware with limited computing power. In addition, many spacecrafts have a shape with multiple symmetries, making an unambiguous initial pose estimation impossible. This work introduces a convolutional-neural-network-based pose estimation method, accounting for potential symmetries of the target satellite. The point clouds are projected to a 2-D depth image before being processed by the network. To generate a sufficient amount of training data, a LiDAR simulator integrating multiple effects such as reflections or laser beam divergence is developed. While being trained solely on synthetic point clouds, the pose estimation method shows to be precise, efficient, and reliable when evaluated on real point clouds taken at a hardware-in-the-loop rendezvous test facility. A runtime evaluation on potential space computing hardware is also performed to demonstrate the applicability of the method to real-time onboard pose estimation.

Existing multi-person video pose estimation methods typically adopt a two-stage pipeline: detecting individuals in each frame, followed by temporal modeling for single-person pose estimation. This design relies on heuristic operations such as tracking, RoI cropping, and non-maximum suppression, limiting both accuracy and efficiency. In this paper, we present a fully end-to-end framework for multi-person 2D pose estimation in videos, effectively eliminating heuristic operations. A key challenge is to associate individuals across frames under complex and overlapping temporal trajectories. To address this, we introduce a novel Pose-Aware Video transformEr Network (PAVE-Net), which features a spatial encoder to model intra-frame relations and a spatiotemporal pose decoder to capture global dependencies across frames. To achieve accurate temporal association, we propose a pose-aware attention mechanism that enables each pose query to selectively aggregate features corresponding to the same individual across consecutive frames. Additionally, we explicitly model spatiotemporal dependencies among pose keypoints to improve accuracy. Notably, our approach is the first end-to-end method for multi-frame 2D human pose estimation. Extensive experiments show that PAVE-Net substantially outperforms prior image-based end-to-end methods, achieving a 6.0 mAP improvement on PoseTrack2017, and delivers accuracy competitive with state-of-the-art two-stage video-based approaches, while offering significant gains in efficiency.

3D object pose estimation by using a depth sensor is one of the important tasks in activities by robots. To reduce the pose ambiguity of an estimated object pose, several methods for multiple viewpoint pose estimation have been proposed. However, these methods need to select the viewpoints carefully to obtain better results. If the pose of the target object is ambiguous from the current observation, we could not decide where we should move the sensor to set as the next viewpoint. In this paper, we propose a best next viewpoint recommendation method by minimizing the pose ambiguity of the object by making use of the current pose estimation result as a latent variable. We evaluated viewpoints recommended by the proposed method and confirmed that it helps us to gain better pose estimation results than several comparative methods on a synthetic dataset.

The widespread adoption of minimally invasive surgical techniques has driven the need for three dimensional (3D) intra-operative image guidance. Hence the pose estimation (position and orientation) performed through the registration of pre-operatively prepared 3D anatomical data to intra-operative two dimensional (2D) fluoroscopy images is one of the main research areas of image guided orthopedic surgery. This paper proposes a non-invasive anatomy-based method for intraoperative pose estimation. The registration is performed solely utilizing bony anatomical features extracted from bi-planar fluoroscopic images (frontal and lateral) without invasive external fiducial markers. The novelty of the proposed methodology is involved in the anatomical features utilized, which are unproblematic and robust to extract and facilitate near real-time pose estimation.

Robotic pollination represents a pivotal component of smart agriculture, with foundational architectures for target recognition, path planning, and motion control having been progressively established. However, developing an efficient and robust pollination system that integrates perception, decision-making, and execution within real-world scenarios remains confronted with complex challenges. This study systematically reviews recent advancements in the field and distills the core technical issues of greenhouse robotic pollination into three primary domains: target detection and pose estimation, end-effector design, and pollination strategies combined with motion control. Focusing on the visual perception of flowers, actuator architecture, and operational tactics, this review synthesizes existing academic findings to evaluate the state-of-the-art in flower detection and pose estimation, characterize diverse end-effector designs, and analyze the evolutionary trajectory of motion control techniques. Specifically, the analysis encompasses the impact of detection algorithms on recognition accuracy and robustness, the structural classification and performance attributes of pollination mechanisms, and the optimization of control strategies. Furthermore, the study categorizes global research backgrounds, technical methodologies, and paradigmatic system cases, offering a critical evaluation of experiences in constructing automated pollination systems. Despite these advances, current robotic pollination technologies for peppers (chili) face significant bottlenecks characterized by immature methods for precise flower detection and pose estimation, the need for optimized specialized end-effector designs, and insufficient robustness in decision-making systems under dynamic environmental conditions. To address these issues, future development should prioritize constructing diverse, large-scale flower image and pose datasets while developing detection algorithms adaptable to complex environments to achieve high-precision identification. Additionally, implementing this system requires a hierarchical architecture where perception drives adaptive actuation. Deep learning models must localize flower targets and assess maturity in real-time, feeding coordinates to path planners that generate collision-free trajectories through foliage. These trajectories are executed via multimodal motion control, synchronizing the rigid manipulator with soft end-effectors. By embedding tactile feedback into the machine learning loop, the system creates a unified sensorimotor framework. This enables dynamic force modulation based on physical resistance, ensuring precise, non-destructive pollination tailored to chili plants.

Pose estimation between cameras and object is a central element for computer vision and its applications. In this paper, we present an approach to solve the problem of estimating the camera 3-D location and orientation from a matched set of 3-D model and 2-D image features. We derive an error equation using roll-pitch-yaw angle to present the rotation matrix and directly calculate the partial derivatives of Jacobian matrix without use of numerical methods for estimation parameters from the nonlinear error equation. Because the proposed method does not use a numerical method to derive the partial derivatives, it is very fast and so adequate for real-time pose estimation and also insensitive to selection of initial values for solving the nonlinear equation. The method is proved from real image experiments and a comparison with a numerical estimation method is presented.

In the context of future manufacturing lines, removing fixtures will be a fundamental step to increase the flexibility of autonomous systems in assembly and logistic operations. Vision-based 3D pose estimation is a necessity to accurately handle objects that might not be placed at fixed positions during the robot task execution. Industrial tasks bring multiple challenges for the robust pose estimation of objects such as difficult object properties, tight cycle times and constraints on camera views. In particular, when interacting with objects, we have to work with close-range partial views of objects that pose a new challenge for typical view-based pose estimation methods. In this paper, we present a 3D pose estimation method based on a gradient-ascend particle filter that integrates new observations on-the-fly to improve the pose estimate. Thereby, we can apply this method online during task execution to save valuable cycle time. In contrast to other view-based pose estimation methods, we model potential views in full 6- dimensional space that allows us to cope with close-range partial objects views. We demonstrate the approach on a real assembly task, in which the algorithm usually converges to the correct pose within 10-15 iterations with an average accuracy of less than 8mm.

Estimating the external calibration --- the pose --- of a camera with respect to its environment is a fundamental task in Computer Vision (CV). In this paper, we propose a novel method for estimating the unknown 6DOF pose of a camera with known intrinsic parameters from epipolar geometry only. For a set of geo-located reference images, we assume the camera position - but not the orientation - to be known. We estimate epipolar geometry between the image of the query camera and the individual reference images using image features. Epipolar geometry inherently contains information about the relative positioning of the query camera with respect to each of the reference cameras, giving rise to a set of relative pose estimates. Combining the set of pose estimates and the positions of the reference cameras in a robust manner allows us to estimate a full 6DOF pose for the query camera. We evaluate our algorithm on different datasets of real imagery in indoor and outdoor environments. Since our pose estimation method does not rely on an explicit reconstruction of the scene, our approach exposes several significant advantages over existing algorithms from the area of pose estimation.