Human Pose Estimation in Videos

We present an approach to solving 2D human pose estimation in videos. The problem of human pose estimation in videos differs from estimating human poses in static images since videos contain a lot of motion related information. Thus, we investigate how to utilize by the information of the human body movements across in a sequence of video frames for estimating human poses in videos. To do this, we introduce a novel heatmap regression method what we call motion-aware heatmap regression. Our approach computes motion vectors in joint keypoints from adjacent frames. We then design a new style of heatmap that we call Motion-Aware Heatmaps to reflect the motion uncertainty of each joint point. Unlike traditional heatmaps, our motion-aware heatmaps not only consider the current joint locations but also account how joints move over time. Furthermore, we introduce a simple yet effective framework designed to incorporate motion information into heatmap regression. We evaluate our motion-aware heatmap regression on PoseTrack(2018, 21) and Sub-JHMDB datasets. Our results validate that the proposed motion-aware heatmaps significantly improve the precision of human pose estimation in videos, particularly in challenging scenarios such as videos like sports game footage with substantial human motions. (Code and related materials are available at https://github.com/Songinpyo/MTPose.)

We present an algorithm for estimating a sequence of articulated upper-body human pose in unconstrained videos. Most previous work often fails to locate forearms in those video scenes suffering from illumination varieties, background clutter, camera shake, or occlusion. In order to deal with such intractable cases, we propose a novel algorithm for addressing the problem of certain body parts localization. The proposed approach can be roughly divided into two steps: first, a spatial model is designed to capture the high-order relationship between adjacent parts and meanwhile to generate a set of configurations in each frame under the temporal context constraint; second, a competitive method is presented to select the best body parts among diverse pose configurations. In this paper, the proposed algorithm focuses on the unconstrained video scenes and improves the detection precision of certain body parts with high degree of freedom. Moreover, the proposed algorithm can be well applied to a very challenging dataset named Movies. Experimental results show that the proposed algorithm can dramatically improve performance compared with those related algorithms on two benchmark datasets (MPII and SHPED datasets) and on our Movies dataset.

Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at https://github.com/zczcwh/PoseFormer

Human pose estimation in videos remains a challenge, largely due to the reliance on extensive manual annotation of large datasets, which is expensive and labor-intensive. Furthermore, existing approaches often struggle to capture long-range temporal dependencies and overlook the complementary relationship between temporal pose heatmaps and visual features. To address these limitations, we introduce STDPose, a novel framework that enhances human pose estimation by learning spatiotemporal dynamics in sparsely-labeled videos. STDPose incorporates two key innovations: 1) A novel Dynamic-Aware Mask to capture long-range motion context, allowing for a nuanced understanding of pose changes. 2) A system for encoding and aggregating spatiotemporal representations and motion dynamics to effectively model spatiotemporal relationships, improving the accuracy and robustness of pose estimation. STDPose establishes a new performance benchmark for both video pose propagation (i.e., propagating pose annotations from labeled frames to unlabeled frames) and pose estimation tasks, across three large-scale evaluation datasets. Additionally, utilizing pseudo-labels generated by pose propagation, STDPose achieves competitive performance with only 26.7% labeled data.

In this paper we consider the problem of human pose estimation in real-world videos of swimmers. Swimming channels allow filming swimmers simultaneously above and below the water surface with a single stationary camera. These recordings can be used to quantitatively assess the athletes' performance. The quantitative evaluation, so far, requires manual annotations of body parts in each video frame. We therefore apply the concept of CNNs in order to automatically infer the required pose information. Starting with an off-the-shelf architecture, we develop extensions to leverage activity information - in our case the swimming style of an athlete - and the continuous nature of the video recordings. Our main contributions are threefold: (a) We apply and evaluate a fine-tuned Convolutional Pose Machine architecture as a baseline in our very challenging aquatic environment and discuss its error modes, (b) we propose an extension to input swimming style information into the fully convolutional architecture and (c) modify the architecture for continuous pose estimation in videos. With these additions we achieve reliable pose estimates with up to +16% more correct body joint detections compared to the baseline architecture.

Pose estimation which is regard as the cross-technology of computer vision and pattern recognition, and an important prerequisite for human behavior understanding. Human pose estimation which use the probability theory, machine learning, pattern recognition, graph theory and other theories to get the position, the deflection angle of the various parts of the body. Then make the detection and estimation parameters for the human body pose. When the image has interference in the background, color and scale changed, human pose complex, self-occlusion and interpersonal interaction occlusion may make the precision and accuracy of pose estimation face great challenge. Thus, according to the above problems, this paper use the advanced model of the human body as contour model to descript the complex pose, in order to make the model more accurate and suitable for various human pose, we preclustering the human body pose of the training samples before we trained the model and in order to ensure the accuracy of the pose we use robust segmentation of multi-view with a novel shape prior. The experiment shows that the algorithm performs better than the classic algorithm on the public datasets.

Human pose estimation (i.e., locating the body parts / joints of a person) is a fundamental problem in human-computer interaction and multimedia applications. Significant progress has been made based on the development of depth sensors, i.e., accessible human pose prediction from still depth images~\cite{rf12pami}. However, most of the existing approaches to this problem involve several components/models that are independently designed and optimized, leading to suboptimal performances. In this paper, we propose a novel inference-embedded multi-task learning framework for predicting human pose from still depth images, which is implemented with a deep architecture of neural networks. Specifically, we handle two cascaded tasks: i) generating the heat (confidence) maps of body parts via a fully convolutional network (FCN); ii) seeking the optimal configuration of body parts based on the detected body part proposals via an inference built-in MatchNet~\cite{mn15cvpr}, which measures the appearance and geometric kinematic compatibility of body parts and embodies the dynamic programming inference as an extra network layer. These two tasks are jointly optimized. Our extensive experiments show that the proposed deep model significantly improves the accuracy of human pose estimation over other several state-of-the-art methods or SDKs. We also release a large-scale dataset for comparison, which includes 100K depth images under challenging scenarios.

3D human pose estimation is a fundamental technology for capturing human motion, providing precise skeletal position information and essential support for fields such as sports analysis, medical rehabilitation, and virtual reality. Recent advances in transformer-based models have demonstrated remarkable success in the domain of 3D human pose estimation. However, existing methods tend to focus predominantly on leveraging spatio-temporal correlations by capturing the inter-frame relationships of individual joints and the identical joint moving across frames, which may not fully harness the complex interactions between different joints across multiple frames. To alleviate this limitation, we introduce a novel SpatioTemporal Aggregation (STA) block designed to capture features based on Spatio-Temporal Aggregation of Pose (STAP). This block adeptly aggregates contextual information from both spatially and temporally proximate joints, effectively modeling the intricate relationships across the entire pose sequence. Technically, the STA block operates by directing the input feature through the two parallel transformation streams, which are tailored to capture the joint-based and STAP-based interactions. The outputs from these transformation layers are then integrated to pay more attention on the dynamic movement patterns of body parts and the overall pose sequence, yielding superior predictive accuracy in 3D pose estimation task. We propose STAPFormer, a model architecture that employs multiple stacked STA blocks, and conduct evaluations on established benchmark datasets. Our model achieves competitive results on the Human3.6M dataset and state-of-the-art performance on the MPI-INF-3DHP dataset, matching existing state-of-the-art methods. In addition, we conduct an ablation study that integrates various configurations of the joint-mixer with the proposed STA block. This study validates the effectiveness of proposed STA block in enhancing 3D human pose estimation. Code and models will soon be available at https://github.com/z2tng/stapformer.

Recently, most existing human pose estimation methods fuse multi-stage convolutional modules to learn a shared feature representation. In this paper, we propose a expectation–maximization (EM) mapping-based network to learn specific related body parts for human pose estimation, named EMposeNet. It maps specific feature of related parts from the original fully shared feature space. From the perspective of multi-task learning, we can regard the task of human pose estimation as a homogeneous multi-task learning. Sharing features among related tasks can result in a more compact model and better generalization ability. However, sharing features for those unrelated or weakly related tasks will deteriorate the estimation performance. Our proposed method aims at performing EM algorithm to learn the related body part, where the predicted keypoint heatmap is potentially more accurate and spatially more precise. We conduct extensive experiments on two benchmark datasets, including the MSCOCO keypoint detection dataset and the MPII human pose dataset, to empirically demonstrate the validity of the proposed method. The results on such two benchmark datasets show that the proposed approach achieves a competitive performance.

Baptizo: A sensor fusion based model for tracking the identity of human poses

International audience

Human pose estimation (HPE) is a field focused on estimating human poses by detecting key points in images. HPE includes methods like top-down and bottom-up approaches. The top-down approach uses a two-stage process, first locating and then detecting key points on humans with bounding boxes, whereas the bottom-up approach directly detects individual key points and integrates them to estimate the overall pose. In this article, we address the problem of bounding box detection inaccuracies in certain situations using the top-down method. The detected bounding boxes, which serve as input for the model, impact the accuracy of pose estimation. Occlusions occur when a part of the target's body is obscured by a person or object and hinder the model's ability to detect complete bounding boxes. Consequently, the model produces bounding boxes that do not recognize occluded parts, resulting in their exclusion from the input used by the HPE model. To mitigate this issue, we introduce the Restoring Occluded Mask Image for 2D Human Pose Estimation (ROM-Pose), comprising a restoration model and an HPE model. The restoration model is designed to delineate the boundary between the target's grayscale mask (occluded image) and the blocker's grayscale mask (occludee image) using the specially created Whole Common Objects in Context (COCO) dataset. Upon identifying the boundary, the restoration model restores the occluded image. This restored image is subsequently overlaid onto the RGB image for use in the HPE model. By integrating occluded parts' information into the input, the bounding box includes these areas during detection, thus enhancing the HPE model's ability to recognize them. ROM-Pose achieved a 1.6% improvement in average precision (AP) compared to the baseline.

We observe that human poses exhibit strong group-wise structural correlation and spatial coupling between keypoints due to the biological constraints of different body parts. This group-wise structural correlation can be explored to improve the accuracy and robustness of human pose estimation. In this work, we develop a self-constrained prediction-verification network to characterize and learn the structural correlation between keypoints during training. During the inference stage, the feedback information from the verification network allows us to perform further optimization of pose prediction, which significantly improves the performance of human pose estimation. Specifically, we partition the keypoints into groups according to the biological structure of human body. Within each group, the keypoints are further partitioned into two subsets, high-accuracy proximal keypoints and low-accuracy distal keypoints. We develop a self-constrained prediction-verification network to perform forward and backward predictions between these keypoint subsets. One fundamental challenge in pose estimation, as well as in generic prediction tasks, is that there is no mechanism for us to verify if the obtained pose estimation or prediction results are accurate or not, since the ground truth is not available. Once successfully learned, the verification network serves as an accuracy verification module for the forward pose prediction. During the inference stage, it can be used to guide the local optimization of the pose estimation results of low-accuracy keypoints with the self-constrained loss on high-accuracy keypoints as the objective function. Our extensive experimental results on benchmark MS COCO and CrowdPose datasets demonstrate that the proposed method can significantly improve the pose estimation results.KeywordsHuman pose estimationSelf-constrainedStructural inferencePrediction optimization

Human Pose Estimation in Video Considering Temporal Consistency

Human Pose Estimation (HPE) is a Computer Vision problem that has become increasingly popular over the last few years, with multiple applications in the medical field such as therapy using virtual and augmented reality, robot caregivers, virtual physical therapy and kinematic analysis. Nevertheless, all the machine learning algorithms developed for these applications are trained in small datasets with images captured in constrained scenarios and with information given by sensors, bounding the applicability of these methods. We developed a simple yet useful deep learning algorithm for Human Pose Estimation that uses as input only an image of a scene with people. The estimated position of the joints and body parts can be used to retrieve basic kinematic information from the people on the image that can be applied to the aforementioned medical applications. We focus on overcoming the limit of Human Pose Estimation algorithms due to jittering, aiming to preserve more precise pixel location. Thus, we explore different novel approaches to improve the precision of the existing state- of-the-art algorithms in keypoint estimation and evaluate them on COCO keypoint dataset, outperforming the current top methods. We hope our algorithm encourages the academic community to develop simpler but precise HPE algorithms for medical applications based on RGB images.