Improved Graph Convolutional Neural Network for Dance Tracking and Pose Estimation

As the basics of 3D vision, pose estimation with 2D images is of significance in 3D reconstruction, UAV positioning, and other fields. However, the related works focus on the natural images and pay less attention to the wide-coverage UAV remote sensing (RS) images. In fact, the relationship between objects in UAV images can benefit pose estimation. Therefore, aiming at the outdoor scene captured by the UAV monocular camera, a novel pose estimation network that emphasizes the association between objects is proposed. The multi-scale visual features extracted by the convolutional neural network (CNN) are manipulated by the object-agnostic segmentation model to indicate the existing space of all possible objects in the whole scene. The features of all possible objects are embedded into vectors, and then processed with a graph convolution network (GCN) for relationship analysis. Based on the known sparse point cloud and the optimized features of 2D images, the camera pose is regressed iteratively by 3D visual geometry. To verify the feasibility of the network, experiments are conducted on the Extended CMU Seasons and the simulation UAV dataset. Results prove that our network emphasizes more features on the small objects and obtains superior pose estimation results.

The intricacies of human motion and action offer a rich vein of information crucial for enhancing the interaction between humans and machines. Human action understanding serves as the linchpin in this dynamic, bridging advanced computational methodologies with real-world applications across various sectors. This dissertation delves into the nuanced realm of action recognition and pose estimation, underscoring the transition from theoretical frameworks to practical applications. Central to the presented research is the development of the Graph Convolution Network (GCN)-based InfoGCN model for human action recognition, which introduces an informa tion bottleneck principle to optimize learning efficiency and accuracy. Additionally, the Pose Relation Transformer (PORT) is crafted to address challenges in hand and human pose esti mation, particularly under conditions of occlusion, by employing strategies akin to Masked Language Modeling in natural language processing. Further advancements are manifested in InfoGCN++, designed to facilitate real-time ac tion recognition, thereby catering to immediate analytical needs in dynamic environments. Concurrently, CARING-AI emerges as a novel framework for generating context-aware hu man motion, capturing both temporal and spatial dimensions to produce more realistic and accurate motion representations. Anticipating future needs, the dissertation proposes CARING-AI++, focusing on the enhancement of hand-object interaction within human motion generation. This progression signifies a deliberate move towards solving user-specific problems, illustrating a profound understanding of the necessary integration of technology and practicality. The collective work outlined in this dissertation not only propels the understanding of human action but also crystallizes its application in real-world scenarios, thereby bridging the gap between theoretical research and practical utility. With implications for healthcare, autonomous vehicles, and interactive technologies, the research underscores the potential of these advancements to revolutionize how we interact with and leverage technology for societal benefit, paving the way for future interdisciplinary collaboration and the ethical use of action recognition systems.

A pose (relative attitude and position) estimation approach to support proximity operations such as autonomous landing of unmanned air vehicles or spacecraft on-orbit servicing is investigated in this paper. An algorithm is developed based on a feature detection capability using the point cloud data that a flash lidar (light detection and ranging) produces. The feature refers to an edge, a corner, or a distinct geometrical shape of an object. Once features are detected, orthogonal triads that are attached to the object are constructed using them. Pose estimation is then performed based on the orientation of the triads with respect to the lidar sensor frame. Since the algorithm employs detected features, it allows pose estimation of an object from a single set of point cloud data. In contrast, the conventional iterative closest point based algorithms typically require two sets of point cloud data. The estimated pose thus represents the changes in pose between the two data sets, leaving the pose of the object with respect to the sensor still to be determined. This featurebased pose estimation capability performs directly an initial pose acquisition with respect to the sensor. The algorithm may also help reduce the number of data points required for pose estimation since a small, selected feature of an object within the sensor field of view can be employed to estimate the pose. Employing the algorithm developed, pose estimation tests were conducted and their preliminary results are presented, which were obtained using a flash lidar in a laboratory testing environment.

In the field of somatosensory game design, traditional human motion recognition methods have limitations. They often overlook the role of the pre-pose estimation algorithm and fail to fully extract motion features, which affects the accuracy and user experience of somatosensory games. This study proposes an improved method for somatosensory game design based on Openpose motion recognition to address this issue. By combining lightweight Openpose and an attention-guided graph convolutional network, it aims to enhance the accuracy and efficiency of the game in recognizing players' motions. This method mainly includes an Openpose algorithm based on ShuffleNet and a graph convolutional algorithm with attention on adjacency matrices at different scales. During the game operation, the input video is processed by the lightweight Openpose to obtain information on 18 human key points and transform it into the form of basic spatio-temporal graph data. Subsequently, the influence of the adjacency matrices corresponding to the neighbor information of nodes at different scales is calculated through the self-attention mechanism. The weighted combination of the adjacency matrices at each scale is input into the graph convolutional network to extract features. The extracted discriminative features are then output as action categories through global average pooling and a Softmax classifier, effectively improving the effect of motion recognition in somatosensory games, bringing more accurate and smooth game experiences to players, possessing high practical value and application prospects, and is expected to promote the further development and improvement of motion recognition technology in somatosensory games.

In recent years, action recognition has gradually developed into a popular research problem in modern computers and other visual science and technology, and has great social science research value. Domestic and foreign experts, scholars and their research units have done a lot of research on this issue, and they have achieved good results. However, the number of research results obtained from the actual application of the human movement recognition technology in the exercise teaching video is relatively small. One of the main reasons is that the motion recognition technology in the human movement teaching video is relatively complex in the application of human movement teaching. At the same time, there is also its own occlusion problem in sports, which leads to a relatively low accuracy of action recognition. Therefore, the development of motion detection and recognition in sports teaching videos is relatively slow. This article aims to study the intelligent recognition of sports teaching videos based on computer vision technology. On the basis of computer vision technology, recognition algorithms and video image preprocessing analysis, support vector machines are used to train sports movements. After the training is completed, the movement can be realized. The experimental results show that the two simulation experiments have achieved good results, verifying the feasibility of the method in the intelligent recognition of sports teaching videos.

In recent years, human motion recognition in computer vision has become a hot research direction in this field. Based on 2D human motion recognition technology, real-time extraction of motion features from 2D planes is used to recognize human movements. This method can only learn the position contour and color information of the image. It cannot directly reflect the motion situation, which results in low recognition accuracy and efficiency. In response to this issue, this study proposes a combination method of motion recognition and 3D pose estimation to recognize and classify basketball movements. First, the 2D skeleton model is obtained by extracting the feature information in the video action, which is converted into a 3D model, and the model is replaced by the time-space convolutional network to establish a human action recognition model. The experiment showed that when the number of iterations reached 6, the accuracy of the spatio-temporal graph convolutional network algorithm model reached 92%. Comparing the accuracy rates of different algorithm models, the average accuracy rates of convolutional neural network, long short memory network, graph convolution, long short model of action recognition and graph convolution model of action recognition were 61.6%, 65.4%, 72.5%, 76.8% and 90.3% respectively. The results show that the proposed 3D recognition algorithm can accurately recognize different basketball movements. This study can provide reference for basketball coaches and athletes in basketball training.

Effective inertial sensor quantity and locations on a body for deep learning-based worker's motion recognition

Advancements in computer vision provide critical insights for various medical use cases, including human behavioral monitoring. Challenges of occlusions and the scarcity of labelled datasets, especially for privacy preserving modalities like long-wave infrared (LWIR) imaging, necessitate innovative approaches for in-bed pose estimation. This study introduces a robust deep learning framework that incorporates novel multimodal feature fusion methodology, generative model, and graph convolutional network (GCN) for in-bed pose estimation. The Simultaneously-collected multimodal Lying Pose (SLP) dataset has been used for performance evaluation of this reported methodology where three modalities, including LWIR, depth, and pressure images, are used to train the model. This model effectively handles modality gaps and adapts to real-world occluded conditions. This work underscores the potential of leveraging multimodal data from privacy-preserving image domains and ability of GCNs to capture spatial joint dependencies for human pose estimation. Extensive evaluations demonstrate the model’s ability to achieve high accuracy of 98.46% at PCKh@0.5 across different cover conditions, outperforming state-of-the-art in-bed pose estimation models on SLP dataset.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Human detection and pose estimation are essential for understanding human activities in images and videos. Mainstream multi-human pose estimation methods take a top-down approach, where human detection is first performed, then each detected person bounding box is fed into a pose estimation network. This top-down approach suffers from the early commitment of initial detections in crowded scenes and other cases with ambiguities or occlusions, leading to pose estimation failures. We propose the DetPoseNet, an end-to-end multi-human detection and pose estimation framework in a unified three-stage network. Our method consists of a coarse-pose proposal extraction sub-net, a coarse-pose based proposal filtering module, and a multi-scale pose refinement sub-net. The coarse-pose proposal sub-net extracts whole-body bounding boxes and body keypoint proposals in a single shot. The coarse-pose filtering step based on the person and keypoint proposals can effectively rule out unlikely detections, thus improving subsequent processing. The pose refinement sub-net performs cascaded pose estimation on each refined proposal region. Multi-scale supervision and multi-scale regression are used in the pose refinement sub-net to simultaneously strengthen context feature learning. Structure-aware loss and keypoint masking are applied to further improve the pose refinement robustness. Our framework is flexible to accept most existing top-down pose estimators as the role of the pose refinement sub-net in our approach. Experiments on COCO and OCHuman datasets demonstrate the effectiveness of the proposed framework. The proposed method is computationally efficient (5-6x speedup) in estimating multi-person poses with refined bounding boxes in sub-seconds.

The research aims to improve the comfort and safety of the smart home by adding a motion recognition algorithm to the smart home system. First, the research status of motion recognition is introduced. Second, based on the requirements of the smart home system, a smart home system is designed for middle-aged and elderly users. The software system in this system includes intelligent control subsystems, intelligent monitoring subsystems, and intelligent protection subsystems. Finally, to increase the security of the smart home, the intelligent monitoring subsystem is improved, and an intelligent security subsystem is proposed based on a small-scale motion detection algorithm. The system uses three three-dimensional (3D) convolutional neural networks (CNNs) to extract three image features, so that the data information in the video can be fully extracted. The performance of the proposed intelligent security subsystem based on a small-scale motion detection algorithm is compared and analyzed. The research results show that the accuracy of the system on the University of Central Florida (UCF101) dataset is 94.64%, and the accuracy on the HMDB51 dataset is 90.11%, which is similar to other advanced algorithms. Observing whether there are dangers such as falling inside and outside the family through motion recognition technology has very important application significance for protecting people's personal safety, life, and health.

With the development of artificial intelligence, human-computer interaction technology has become more and more popular. Motion capture and recognition technology based on sensor networks and machine learning algorithms gradually drawn wide attention and extensive works have been done. This article introduces the arm motion capture system based on MEMS sensor networks that takes the micro inertial measurement unit as the core, and uses the Kernel Perceptron Algorithm (KPA) to realize motion classification and recognition. This algorithm combines the advantages of Support Vector Machine (SVM) and perceptron algorithm to gain a faster model training speed under the premise of high recognition accuracy. Extensive experiments can prove that arm motions can be accurately captured by MEMS sensor networks, and the KPA has a good performance of motion recognition and faster model training speed than SVM.

Human action identification has advanced significantly as a result of the development of deep learning algorithms. Convolutional Neural Networks (CNNs) are known for their adeptness at extracting crucial information within video frames, especially shapes and postures. However, understanding movements necessitates not just individual frames as well as the temporal flow of the movement; this is wherein LSTM (long short-term memory) variants of Recurrent Neural Networks (RNNs) come into play. By analyzing frame sequences, LSTMs are particularly effective in gathering temporal connections, recalling prior motions, and overcoming the gap amongst static features as well as dynamic actions. Even with CNNs and LSTMs' remarkable accuracy in human action recognition (HAR), challenges remain.

Summary Oilfield development performance prediction is a significant and complex problem in oilfield development. Reasonable prediction of oilfield development performance can guide the adjustment of the development plan. Moreover, the reservoir will change slowly during reservoir development because of flowing water however, previous networks that forecast production dynamics ignored it, which leads to inaccurate predictions. Routine well-wise injection and production measurements contain important subsurface structure and properties. So, for the dynamic prediction of oil/water two-phase waterflooded reservoirs, we built a deep learning framework named adaptive correction interwell connectivity model based on graph convolutional networks (GCN) and gated recurrent unit (GRU). It includes two parts: The first part is the adaptive correction model based on GCN, which uses dynamic production data to automatically correct the initial interwell connectivity computed by permeability, porosity, interwell distance, and so on. The second part is the adaptive learning model based on GRU, which predicts the production performance of oil wells according to the time characteristics of production performance data. This framework considers the influence that changes in reservoir conditions have on production over time to solve the problem of inaccurate production dynamic prediction. It can also predict interwell connectivity. For oilfields with too many wells, using the embedding idea classifies similar wells into one category, saving time for training and avoiding overfitting problems. Applying the model to five different reservoirs to predict interwell connectivity, well oil production rate, and well water cut compare the results with artificial neural networks (ANN), GRU, and long short-term memory (LSTM) models and compare the interwell connectivity with numerical simulation software ,tNavigator® (Rock Flow Dynamics Llc), too. When the model is applied in Block B of Bohai A reservoir, the mean absolute percentage error of “Adaptive Graph convolutional network and GRU” (AG-GRU) is 2.1150% while the LSTM is 9.8872%. The error reduces by 78.6%. The injected water has a direction from the water injection well to the production well; this paper only considers the interwell connectivity without considering the direction. Further research is needed to consider the water injection direction and form a weighted directed graph.

Accurate prediction of surface subsidence is of significance for analyzing the pattern of mining-induced surface subsidence, and for mining under buildings, railways, and water bodies. To address the problem that the existing prediction models ignore the correlation between subsidence points, resulting in large prediction errors, a Multi-point Relationship Fusion prediction model based on Graph Convolutional Networks (MRF-GCN) for mining-induced subsidence was proposed. Taking the surface subsidence in 82/83 mining area of Yuandian No. 2 Mine in Anhui Province in eastern China as an example, the surface deformation data obtained from 250 InSAR images captured by Sentinel-1A satellite from 2018 to 2022, combined with GNSS observation data, were used for modeling. The deformation pattern of each single observation point was obtained by feeding their deformation observation data into the LSTM encoder, after that, the relationship graph was created based on the correlation between points in the observation network and MRF-GCN was established. Then the prediction results came out through a nonlinear activation function of neural network. The research shows that the R2R2 value of MRF-GCN model was 0.865 0, much larger than that of Long-Short Term Memory (LSTM) and other conventional models, while mean square error (MSE) of MRF-GCN model was 1.59 899, much smaller than that of LSTM and other conventional models. Therefore, the MRF-GCN model has better prediction accuracy than other models and can be applied to predicting surface subsidence in large areas.