3D human pose estimation from image using couple sparse coding

Automatically estimating 3D human poses in video and inferring their meanings play an essential role in many human-centered automation systems. Existing researches made remarkable progresses by first estimating 2D human joints in video and then reconstructing 3D human pose from the 2D joints. However, mono-directionally reconstructing 3D pose from 2D joints ignores the interaction between information in 3D space and 2D space, losses rich information of original video, therefore limits the ceiling of estimation accuracy. To this end, this paper proposes a bidirectional 2D-3D transformation framework that bidirectionally exchanges 2D and 3D information and utilizes video information to estimate an offset for refining 3D human pose. In addition, a bone-length stability loss is utilized for the purpose of exploring human body structure to make the estimated 3D pose more natural and to further increase the overall accuracy. By evaluation, estimation error of the proposed method, measured by the mean per joint position error (MPJPE), is only 46.5 mm, which is much lower than state-of-the-art methods under the same experimental condition. The improvement on accuracy will make machines to better understand human poses for building superior human-centered automation systems. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the demand of human-centered automation systems needing to accurately understand human poses. Existing approaches mainly focus on inferring 3D human pose from 2D joints mono-directionally. Although they made remarkable contributions to estimating 3D human pose in such a mono-directional way, we found that they ignore the 2D-3D interaction and do not use original video when inferring 3D pose from 2D joints. This paper therefore suggests a bidirectional 2D-3D transformation that exchanges 2D and 3D information and utilizes video information to estimate more accurate 3D human pose for human-centered automation systems. This work is a pioneering attempt of interactively using 2D and 3D information for more accurate estimation of human pose. Benefited from the state-of-the-art accuracy, the proposed approach is expected to make significant contributions to many human-centered automation systems, such as human-machine interaction, biomimetic manipulation, and automatic surveillance systems.

3D human pose estimation aims to reconstruct the human skeleton of all the individuals in a scene by detecting several body joints. The creation of accurate and efficient methods is required for several real-world applications including animation, human–robot interaction, surveillance systems or sports, among many others. However, several obstacles such as occlusions, random camera perspectives, or the scarcity of 3D labelled data, have been hampering the models’ performance and limiting their deployment in real-world scenarios. The higher availability of cameras has led researchers to explore multi-view solutions due to the advantage of being able to exploit different perspectives to reconstruct the pose. Most existing reviews focus mainly on monocular 3D human pose estimation and a comprehensive survey only on multi-view approaches to determine the 3D pose has been missing since 2012. Thus, the goal of this survey is to fill that gap and present an overview of the methodologies related to 3D pose estimation in multi-view settings, understand what were the strategies found to address the various challenges and also, identify their limitations. According to the reviewed articles, it was possible to find that most methods are fully-supervised approaches based on geometric constraints. Nonetheless, most of the methods suffer from 2D pose mismatches, to which the incorporation of temporal consistency and depth information have been suggested to reduce the impact of this limitation, besides working directly with 3D features can completely surpass this problem but at the expense of higher computational complexity. Models with lower supervision levels were identified to overcome some of the issues related to 3D pose, particularly the scarcity of labelled datasets. Therefore, no method is yet capable of solving all the challenges associated with the reconstruction of the 3D pose. Due to the existing trade-off between complexity and performance, the best method depends on the application scenario. Therefore, further research is still required to develop an approach capable of quickly inferring a highly accurate 3D pose with bearable computation cost. To this goal, techniques such as active learning, methods that learn with a low level of supervision, the incorporation of temporal consistency, view selection, estimation of depth information and multi-modal approaches might be interesting strategies to keep in mind when developing a new methodology to solve this task. • First review only on multi-view, multi-modal methods to estimate 3D pose since 2012. • Multi-view allows capturing the full body geometry, making 3D pose estimation easier. • Real-world applications include sports, broadcasting, rehabilitation or animation. • Finding a fast, accurate method with low computational cost remains a challenge. • Multi-modal methods or view selection can lead to an efficient and effective model.

In the field of 3D Human Pose Estimation and Reconstruction based on body joints extracted from a 2D image; Exists challenges like self-occlusion and depth perception. These problems hinder approximate estimations. This article proposes a hybrid method that consists of semantic segmentation, sparse representation and 2D pose estimation for 3D estimation. Specifically, we train two fully Convolution neural networks (FCNs) to estimate 2D pose and semantic segmentation. Then, we narrow the estimation down using basic human body structure and results from the FCNs. Subsequently, utilizing the estimated 2D pose of the earlier step and a sparse representation model the 3D pose will estimate. Using a Convolution Neural Networks learning method for 2D human pose estimation and then by sparse representation will estimate. Results of the proposed method, show improvement relative to the previous works on 3D Human pose estimation. 3D Human Pose Estimation by this method demonstrates a significant mean error reduction compared to earlier studies.

SkeletonPose: Exploiting human skeleton constraint for 3D human pose estimation

Weakly-supervised pre-training for 3D human pose estimation via perspective knowledge

Human pose estimation from video sequences has become a hot research topic in the domain of robotics and computer vision. However, existing three-dimensional (3D) pose estimation methods usually analyze individual frames, which have a low accuracy due to various human movement speed, limiting its practical application. In this paper, we propose a method for estimating 3D pose and calculating similarity from Tai Chi video sequences based on Seq2Seq network. Specifically, using 2D joint point coordinate sequence of the original image as input, our method constructs an encoder and a decoder to build a Seq2Seq network. Our method introduces an attention mechanism for weighing the input data to obtain an intermediate vector and decode it to estimate the 3D joint point sequence. Afterwards, using a template video and a target video as input, our method calculates the cost of passing through each point within the constraints to construct a cost matrix for video similarity. With the cost matrix, our method can determine the optimal path and use the correspondence of the video sequence to calculate the image similarity of the corresponding frame. The experimental data show that the proposed method can effectively improve the accuracy of 3D pose estimation, and increase the speed for video similarity calculation.

The estimation of 3D body pose and shape has always been a challenging problem due to various reasons, such as the ambiguity in 2D images and complex articulated structure of the human body. In order to solve the ill-conditioned problems, in this paper, we bring up an end-to-end method to estimate 3D human shape and pose from multi-view RGB images. In the proposed framework, we first implement a CNN embedded with attention module to extract the image feature and design the view-pooling layer to combine the features from multiple views. Then we adopt a regression network with a novel geometric constraint of body limbs to estimate 3D human pose and shape. Additionally, during the training process, we employ the idea of adversarial learning in our model to help regress accurate pose and shape parameters. Extensive experiments are conducted on Human3.6M and MPI-INF-3DHP datasets, and our method achieves competitive results in the 3D pose and shape estimation task.

We present a deployment friendly, fast bottom-up framework for multi-person 3D human pose estimation. We adopt a novel neural representation of multi-person 3D pose which unifies the position of person instances with their corresponding 3D pose representation. This is realized by learning a generative pose embedding which not only ensures plausible 3D pose predictions, but also eliminates the usual keypoint grouping operation as employed in prior bottom-up approaches. Further, we propose a practical deployment paradigm where paired 2D or 3D pose annotations are unavailable. In the absence of any paired supervision, we leverage a frozen network, as a teacher model, which is trained on an auxiliary task of multi-person 2D pose estimation. We cast the learning as a cross-modal alignment problem and propose training objectives to realize a shared latent space between two diverse modalities. We aim to enhance the model’s ability to perform beyond the limiting teacher network by enriching the latent-to-3D pose mapping using artificially synthesized multi-person 3D scene samples. Our approach not only generalizes to in-the-wild images, but also yields a superior trade-off between speed and performance, compared to prior top-down approaches. Our approach also yields state-of-the-art multi-person 3D pose estimation performance among the bottom-up approaches under consistent supervision levels.

The neural network based approach for 3D human pose estimation from monocular images has attracted growing interest. However, annotating 3D poses is a labor-intensive and expensive process. In this paper, we propose a novel self-supervised approach to avoid the need of manual annotations. Different from existing weakly/self-supervised methods that require extra unpaired 3D ground-truth data to alleviate the depth ambiguity problem, our method trains the network only relying on geometric knowledge without any additional 3D pose annotations. The proposed method follows the two-stage pipeline: 2D pose estimation and 2D-to-3D pose lifting. We design the transform re-projection loss that is an effective way to explore multi-view consistency for training the 2D-to-3D lifting network. Besides, we adopt the confidences of 2D joints to integrate losses from different views to alleviate the influence of noises caused by the self-occlusion problem. Finally, we design a two-branch training architecture, which helps to preserve the scale information of re-projected 2D poses during training, resulting in accurate 3D pose predictions. We demonstrate the effectiveness of our method on two popular 3D human pose datasets, Human3.6M and MPI-INF-3DHP. The results show that our method significantly outperforms recent weakly/self-supervised approaches.

To deal with the problem where each instance is associated with multiple labels, a lot of multi-label learning algorithms have been developed in recent years. Some approaches have been proposed to select label-specific features to utilize discriminate features for multi-label classification. Although label correlation has been considered in learning label-specific features, the critical correlation among instances was less taken into account. In this paper, we proposed a new approach called multi-label learning with label-specific features using correlation information (LSF-CI) to learn label-specific features for each label with the consideration of both correlation information in label space and correlation information in feature space. In the LSF-CI, the instance correlation in feature space is computed by a probabilistic neighborhood graph model, and label correlation in label space is computed by cosine similarity. For multi-label data, the LSF-CI has the capability to select Label-specific features for each label as well as classify an unseen instance into a set of relevant labels. To validate the effectiveness of LSF-CI, we conducted comprehensive experiments on eight multi-label datasets. The experimental results demonstrate that the LSF-CI is capable of selecting compact label-specific features, and achieving a competitive performance in comparison with the performances of the existing multi-label learning approaches.

Label propagation via local geometry preserving for deep semi-supervised image recognition

Enhancement of human 3D pose estimation using a novel concept of depth prediction with pose alignment from a single 2D image

Face pose information is valuable for a variety of applications including unconstrained face recognition, natural human computer interfaces, and video database indexing. 3D pose estimation is a critical requirement for accurate face recognition using view varying representations, such as 2D intensity images. 3D pose extraction in this context requires 3D information, which is present in image sequences if we assume the moving objects in the sequence are primarily rigid. This paper presents a motion based pose estimation system which computes the 3D pose of the head in each frame of a video sequence. Generic low level features, such as corners, are identified and tracked in the video stream. The feature tracks are processed by a shape from motion algorithm which produces estimates of 3D geometry and pose. The geometry and pose estimates are considered together with facial structure constraints, temporal constraints, and initial pose estimates to refine knowledge of the specific face structure and its pose. We describe this system and show that it works on human faces. This is significant because the face has many smooth surfaces which make it difficult to extract dense intensity features.

The accuracy of monocular 3D human pose estimation depends on the viewpoint from which the image is captured. While freely moving cameras, such as on drones, provide control over this viewpoint, automatically positioning them at the location which will yield the highest accuracy remains an open problem. This is the problem that we address in this paper. Specifically, given a short video sequence, we introduce an algorithm that predicts which viewpoints should be chosen to capture future frames so as to maximize 3D human pose estimation accuracy. The key idea underlying our approach is a method to estimate the uncertainty of the 3D body pose estimates. We integrate several sources of uncertainty, originating from deep learning based regressors and temporal smoothness. Our motion planner yields improved 3D body pose estimates and outperforms or matches existing ones that are based on person following and orbiting.

3D hand pose estimation can provide basic information about gestures, which has an important significance in the fields of Human-Machine Interaction (HMI) and Virtual Reality (VR). In recent years, 3D hand pose estimation from a single depth image has made great research achievements due to the development of depth cameras. However, 3D hand pose estimation from a single RGB image is still a highly challenging problem. In this work, we propose a novel four-stage cascaded hierarchical CNN (4CHNet), which leverages hierarchical network to decompose hand pose estimation into finger pose estimation and palm pose estimation, extracts separately finger features and palm features, and finally fuses them to estimate 3D hand pose. Compared with direct estimation methods, the hand feature information extracted by the hierarchical network is more representative. Furthermore, concatenating various stages of the network for end-to-end training can make each stage mutually beneficial and progress. The experimental results on two public datasets demonstrate that our 4CHNet can significantly improve the accuracy of 3D hand pose estimation from a single RGB image.