Cross-Modal Vision-Language Model for High-Precision Human Pose Estimation

Human Pose Estimation (HPE) is defined as the problem of human joints’ localization (also known as keypoints: elbows, wrists, etc.) in images or videos. It is also defined as the search for a specific pose in space of all articulated joints. HPE has recently received significant attention from the scientific community. The main reason behind this trend is that pose estimation is considered as a key step for many computer vision tasks. Although many approaches have reported promising results, this domain remains largely unsolved due to several challenges such as occlusions, small and barely visible joints, and variations in clothing and lighting. In the last few years, the power of deep neural networks has been demonstrated in a wide variety of computer vision problems and especially the HPE task. In this context, we present in this paper a Deep Full-Body-HPE (DFB-HPE) approach from RGB images only. Based on ConvNets, fifteen human joint positions are predicted and can be further exploited for a large range of applications such as gesture recognition, sports performance analysis, or human-robot interaction. To evaluate the proposed deep pose estimation model, we apply it to recognize the daily activities of a person in an unconstrained environment. Therefore, the extracted features, represented by deep estimated poses, are fed to an SVM classifier. To validate the proposed architecture, our approach is tested on two publicly available benchmarks for pose estimation and activity recognition, namely the J-HMDBand CAD-60datasets. The obtained results demonstrate the efficiency of the proposed method based on ConvNets and SVM and prove how deep pose estimation can improve the recognition accuracy. By means of comparison with state-of-the-art methods, we achieve the best HPE performance, as well as the best activity recognition precision on the CAD-60 dataset.

In the field of 3D Human Pose Estimation and Reconstruction based on body joints extracted from a 2D image; Exists challenges like self-occlusion and depth perception. These problems hinder approximate estimations. This article proposes a hybrid method that consists of semantic segmentation, sparse representation and 2D pose estimation for 3D estimation. Specifically, we train two fully Convolution neural networks (FCNs) to estimate 2D pose and semantic segmentation. Then, we narrow the estimation down using basic human body structure and results from the FCNs. Subsequently, utilizing the estimated 2D pose of the earlier step and a sparse representation model the 3D pose will estimate. Using a Convolution Neural Networks learning method for 2D human pose estimation and then by sparse representation will estimate. Results of the proposed method, show improvement relative to the previous works on 3D Human pose estimation. 3D Human Pose Estimation by this method demonstrates a significant mean error reduction compared to earlier studies.

Human pose estimation (i.e., locating the body parts / joints of a person) is a fundamental problem in human-computer interaction and multimedia applications. Significant progress has been made based on the development of depth sensors, i.e., accessible human pose prediction from still depth images~\cite{rf12pami}. However, most of the existing approaches to this problem involve several components/models that are independently designed and optimized, leading to suboptimal performances. In this paper, we propose a novel inference-embedded multi-task learning framework for predicting human pose from still depth images, which is implemented with a deep architecture of neural networks. Specifically, we handle two cascaded tasks: i) generating the heat (confidence) maps of body parts via a fully convolutional network (FCN); ii) seeking the optimal configuration of body parts based on the detected body part proposals via an inference built-in MatchNet~\cite{mn15cvpr}, which measures the appearance and geometric kinematic compatibility of body parts and embodies the dynamic programming inference as an extra network layer. These two tasks are jointly optimized. Our extensive experiments show that the proposed deep model significantly improves the accuracy of human pose estimation over other several state-of-the-art methods or SDKs. We also release a large-scale dataset for comparison, which includes 100K depth images under challenging scenarios.

Detecting human pose in a video is a difficult task. Although many high-performed human pose estimation models have been proposed in the last few years, the estimation accuracy has always been a major concern. In this study we present a method to improve the accuracy of human pose estimation for videos. Technically, predicted human pose is a set of time series data. Thus, by using time series correlation, human pose estimation can be performed in a better accuracy. We combine a CNN based human pose estimation model with a multiple object tracking framework to achieve this. Undetected/mis-detected body joints will be interpolated using the information from previous and following frames. As a result, our proposed method improved the accuracy of an existing CNN based human pose estimation model by reducing the number of undetected and mis-detected frames by 6.30% and 0.98% respectively.

In computer vision, human pose estimation and the following action recognition are active research topics, which have a wide application in human computer interaction, smart homes, athlete assistance training, etc. Driven by these applications, numerous algorithms and models are proposed in the recent years, among which deep learning becomes dominant. In this study, we summarize the recent progress of deep learning in human pose estimation and action recognition. We also point out some research challenges and future research directions.

It is widely shown that Convolutional Neural Networks (CNNs) are vulnerable to adversarial examples on most recognition tasks, such as image classification and segmentation. However, few work studies the more complicated task – 3D human pose estimation. This task often requires large-scale datasets, specialized network architectures, and it can be solved either from single-view RGB images or from multi-view RGB images. In this paper, we make the first attempt to investigate the robustness of current state-of-the-art 3D human pose estimation methods. To this end, we build four representative baseline models, where most of the current methods can be generally classified as one of them. Furthermore, we design targeted adversarial attacks to detect whether 3D pose estimators are robust to different camera parameters. For different types of methods, we present a comprehensive study of their robustness on the large-scale Human3.6M benchmark. Our work shows that different methods vary significantly in their resistance to adversarial attacks. Through extensive experiments, we show that multi-view 3D pose estimators can be more vulnerable to adversarial examples. We believe that our efforts can shed light on future works to design more robust 3D human pose estimators.

Human pose estimation (HPE) is most dominant research areas in Computer Vision (CV) field. This technology will have huge implications. The significant applications include human computer interaction, activity recognition, video surveillance, motion recognition, etc. Different types of pose estimation are used for estimating the number of persons who are tracked. In this paper we are focusing on single 2D pose estimation. Present trends of pose estimation uses CNN based architectures for HPE and post statistical methods. In this work we propose a monadic frame work for both HPE and post processing into a single stage. It produces a human pose skeleton for a single two-dimensional image and videos. Here, take different modalities such as static image, static video and live video to generate different skeletons for SHPE. In this work we use pre-trained tensor flow based CNN model for pose estimation and also it produce better results compared to state of art in terms of performance metrics as PCP@.5, PCK@.5, and PCKh@.2.

This letter proposes an accurate human pose estimation method that uses a modified kernel density approximation (m-KDA) to multiple pose hypotheses. Existing methods show poor human pose estimation because of cluttered background or self-occlusion by the human. To improve the pose estimation accuracy, we propose to use m-KDA to aggregate multiple pose estimation results. First, we use the flexible mixture-of-parts model (FMM) to estimate the human poses then use the top-M scores to choose the good pose hypotheses. Second, we aggregate the top-M pose hypotheses with the m-KDA, in which each kernel density function is modified by each pose's score value and each pose's compatibility function that represents how far each pose hypothesis is departed from the nominal value of top-M pose hypotheses. Third, we determine the optimal pose configuration by repeating the above m-KDA computation, starting from the root part (head) to the leaf parts (hands and feet), sequentially. In pose estimation experiments on two benchmark datasets (PARSE and LSP), the proposed method achieved 1.5-4.0% improvement in the percentage of correct localized parts (PCP) over the state-of-the-art methods.

Validity of an artificial intelligence, human pose estimation model for measuring single-leg squat kinematics

Articulated human pose estimation is a fundamental yet challenging task in computer vision. The difficulty is particularly pronounced in scale variations of human body parts when camera view changes or severe foreshortening happens. Although pyramid methods are widely used to handle scale changes at inference time, learning feature pyramids in deep convolutional neural networks (DCNNs) is still not well explored. In this work, we design a Pyramid Residual Module (PRMs) to enhance the invariance in scales of DCNNs. Given input features, the PRMs learn convolutional filters on various scales of input features, which are obtained with different subsampling ratios in a multibranch network. Moreover, we observe that it is inappropriate to adopt existing methods to initialize the weights of multi-branch networks, which achieve superior performance than plain networks in many tasks recently. Therefore, we provide theoretic derivation to extend the current weight initialization scheme to multi-branch network structures. We investigate our method on two standard benchmarks for human pose estimation. Our approach obtains state-of-the-art results on both benchmarks. Code is available at https://github.com/bearpaw/PyraNet.

Human pose estimation is a challenging visual task that relies on spatial location information. To improve the performance of human pose estimation, it is important to accurately determine the constraint relationship among keypoints. To address this, we propose MfvPose, a novel hybrid model that leverages rich multi-scale information. The proposed model incorporates the HRFOV module, which uses cascaded atrous convolution to maintain high-resolution representations of the backbone extractor and enrich the multi-scale information. In addition, we introduce learnable scalar weights to the Transformer encoder. In detail, it involves a multiplication by a diagonal matrix with learnable scalar weights on output of each residual block, which improves the dynamics of model training and enhances the accuracy of human pose estimation. It is experimentally shown that our proposed MfvPose achieves promising results on various benchmarks.

Human pose estimation and human activity recognition are two of the most researched domains in computer vision applications. Surveillance, human-computer interaction and video retrieval, all benefit from robust solutions of this domain. Due to disparities in inter-personal parameters, mobility efficiency, and recording configurations, the process is challenging. In this paper, deep learning algorithms are used for both pose estimation and activity recognition models. The pose estimation model generates an array of objects for each keypoint with their coordinated over a specific interval. This time series data is then consumed by the activity recognition model which was trained using a temporal convolutional neural network. The attained accuracy of the model was found to be 92.7% with a loss of 0.19.

In recent years, Convolutional Neural Networks (CNNs) have been widely successful for numerous computer vision tasks using 2D color and 3D point data, such as object detection and classification, and 3D pose estimation. Despite these successes, it comes with a price --- CNNs require a vast amount of labeled training data to be successful. Many training datasets are now available, however, they are mostly limited to general objects in the public domain such as people, cars, buildings, fruit, and more. Thus, not all fields have sufficient training data. Even if an adequate number of data samples can be built into a dataset, manually collecting and labeling the training data remains a laborious task. Several approaches to reduce or eradicate the need for training data have been proposed by researchers. The two most popular approaches are training with synthetic data and training on generalized object features. Training with synthetic data refers to the use of computer graphics to generate training data. While this approach has demonstrated success, it comes with its own challenges. In contrast, training a CNN on generalized object features identifies specific local 3D features or control points on objects. Local 3D features and their descriptors have proven in the past to be universally applicable and effective for general object detection, and thus, this method allows for a generalized CNN. A generalized CNN refers to a network architecture that, after initial training, can be specialized for a new task with a minimum amount or no new training data. This dissertation investigates an encoder-decoder architecture and determines the capabilities of it to generalize the CNN architecture for object detection tasks in point cloud data. A CNN is trained to recognize 3D features and demonstrates, without retraining, its successful applicability for point cloud object detection using disparate data. Additionally, the geometric consistency of the descriptors is evaluated to obtain further insight. The results demonstrate that the proposed 3D feature descriptors, and their increased geometric consistency contribute to the increased encoder-decoder architecture performance. In summary, this dissertation contributes a state-of-the-art, generalized, feature descriptor-based CNN architecture that can be transferred to different objects without retraining. Furthermore, it provides insight explaining the increased performance of the CNN compared to the state-of-the-art.

Radio-frequency (RF) based human sensing technologies, due to their great practical value in various applications and privacy-preserving nature, have gained tremendous attention in recent years. However, without fully exploiting the characteristics of radio signals, the performance of existing methods are still limited. First, RF features of the moving human body have different representations in dimensions such as channel and scale, which is challenging when performing feature fusion. Besides, the human body is specularly reflective with respect to the radar, which means the human body cannot be fully captured by a single RF snapshot. Therefore, the radar signal reflected by the human body is sparse and incomplete, which is difficult to extract high-quality features for 3D human pose estimation. In this paper, we present the RF-based Pose Machines (RPM), a novel framework which can generate 3D skeletons from RF signals. Considering the characteristics of RF signals, RPM includes several modules to overcome the challenges. Firstly, a Multidimensional Feature Fusion (MFF) backbone is designed to effectively fuse radio signals based on the channels' correlation and maintain high-quality feature via a multi-scale fusion block. A Spatio-Temporal Attention network is then designed to reconstruct 3D skeletons by modeling the non-local spatio-temporal relationships. To evaluate the performance of our RPM framework, we construct a large-scale dataset of synchronized 3D skeletons and RF signals, RFSkeleton3D. Our experimental results show that RPM locates 3D key points of the human body with an average error of 5.71cm and maintains its performance in new environments with occlusion or bad illumination. The dataset and codes will be made in public.

Human pose estimation is an important downstream task in computer vision, with significant applications in action recognition and virtual reality. However, data collected in a decentralized manner often exhibit non-independent and identically distributed (non-IID) characteristics, and traditional federated learning aggregation strategies can lead to gradient conflicts that impair model convergence and accuracy. To address this, we propose the Federated Gradient Harmonization aggregation strategy (FedGH), which coordinates update directions by measuring client gradient discrepancies and integrating gradient-projection correction with a parameter-reconstruction mechanism. Experiments conducted on a self-constructed single-arm robotic dataset and the public Max Planck Institute for Informatics (MPII Human Pose Dataset) dataset demonstrate that FedGH achieves average Percentage of Correct Keypoints (PCK) of 47.14% and 66.31% across all keypoints, representing improvements of 1.82 and 0.36 percentage points over the Federated Adaptive Weighting (FedAW) method. On our self-constructed dataset, FedGH attains a PCK of 86.4% for shoulder detection, surpassing other traditional federated learning methods by 20–30%. Moreover, on the self-constructed dataset, FedGH reaches over 98% accuracy in the keypoint heatmap regression model within the first 10 rounds and remains stable between 98% and 100% thereafter. This method effectively mitigates gradient conflicts in non-IID environments, providing a more robust optimization solution for distributed human pose estimation.