DeepVI: A Novel Framework for Learning Deep View-Invariant Human Action Representations using a Single RGB Camera

The main core purpose of artificial emotional intelligence is to recognize human emotions. Technologies such as facial, semantic, or brainwave recognition applications have been widely proposed. However, the abovementioned recognition techniques for emotional features require a large number of training samples to obtain high accuracy. Human behaviour pattern can be trained and recognized by the continuous movement of the Spatial Temporal Graph Convolution Network (ST-GCN). However, this technology does not distinguish between the speed of delicate emotions, and the speed of human behaviour and delicate changes of emotions cannot be effectively distinguished. This research paper proposes Spatial Temporal Variation Convolutional Network training for human emotion recognition, using skeleton detection technology to calculate the degree of skeleton point change between consecutive actions and using the nearest neighbour algorithm to classify speed levels and train the ST-GCN recognition model to obtain the emotional state. Application of the speed change recognition ability of the Spatial Temporal Variation Graph Convolution Network (STV-GCN) to artificial emotional intelligence calculation makes it possible to efficiently recognize the delicate actions of happy, sad, fear, and angry in human behaviour. The STV-GCN technology proposed in this paper is compared with ST-GCN and can effectively improve the recognition accuracy by more than 50%.

In autonomous driving scenarios, pedestrian trajectory prediction is an important research direction. Based on the spatio-temporal graph convolutional neural network, we propose a new pedestrian trajectory prediction algorithm. The new algorithm constructs a series of new models around pedestrian intention estimation. The construction of the estimation algorithm considers the following aspects: the contextual information of pedestrians and the surrounding environment, the “pedestrian ego-vehicle” interaction combined with the vehicle speed estimation, the pedestrian’s own skeletal structure information and body language estimation, which includes head joints and the relative structural relationship of the torso joints, including whether it is out of the same plane, is rotated, and so on. Skeleton information feature extraction and construction adopts the method of graph convolutional neural network to structure pedestrians into joints in the form of graphs in non-Euclidean space, and further adopts spatial temporal graph convolutional network for feature extraction and learning. The new method is named a “head-torso”-based spatial temporal graph convolutional network (HT-STGCN). On the dataset PID, the novel method achieves substantial improvements over mainstream methods. Experimental results show that combining HT-STGCN with observed action can improve trajectory prediction.

In recent years, the increasing labor burden in Japanese agriculture hs become a serious issue, driving the development of robots to assist in transporting harvested crops. This study proposes a method that recognizes the action transitions of agricultural workers and enables smooth transport assistance by utilizing three-dimensional skeletal information obtained from RGB-D images. Specifically, we employ Spatial Temporal Graph Convolutional Networks (ST-GCN) to detect the transition from "harvesting" to "loading." The recognition results are used to control the robot so that it approaches the worker before the loading action begins. The proposed method introduces a new labeling scheme tailored to harvesting and crop-loading motions, thereby improving recognition performance with ST-GCN. Evaluation experiments verified its generalization capability to different harvesting postures and workers, demonstrating an 18.7% improvement in action transition recognition accuracy compared with conventional methods. Furthermore, in robot-following experiments with the proposed method implemented, we confirmed that the system could both recognize action transitions and adjust the target following distance before the worker started loading. These results show that an ST-GCN specialized for agricultural tasks can effectively recognize harvesting action transitions, contributing to reducing the burden on workers during crop transportation.

Using complex networks and multiple artificial intelligence algorithms for table tennis match action recognition and technical-tactical analysis

Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.

We tackle the problem of tracking the human lower body as an initial step toward an automatic motion assessment system for clinical mobility evaluation, using a multimodal system that combines Inertial Measurement Unit (IMU) data, RGB images, and point cloud depth measurements. This system applies the factor graph representation to an optimization problem that provides 3-D skeleton joint estimations. In this paper, we focus on improving the temporal consistency of the estimated human trajectories to greatly extend the range of operability of the depth sensor. More specifically, we introduce a new factor graph factor based on Koopman theory that embeds the nonlinear dynamics of several lower-limb movement activities. This factor performs a two-step process: first, a custom activity recognition module based on spatial temporal graph convolutional networks recognizes the walking activity; then, a Koopman pose prediction of the subsequent skeleton is used as an a priori estimation to drive the optimization problem toward more consistent results. We tested the performance of this module a dataset composed of multiple clinical lower-limb mobility tests, and we show that our approach reduces outliers on the skeleton form by almost 1 m, while preserving natural walking trajectories at depths up to more than 10 m.

Three-dimensional (3D) medical images are prone to overlap, and there are some problems, such as low detection efficiency and inconsistent with the actual situation. Therefore, a 3D medical image surface reconstruction method based on data mining and machine learning is proposed. The 3D medical images were classified according to different ways, the information frame of 3D medical images was established and the surface overlapping information model of 3D images was given. Based on this information framework, the nonlinear function of overlapping area information of 3D medical images was constructed. The weight of the nonlinear function was used to calculate the input and output results of overlapping area information. Combined with the input mode of 3D medical image information, the error between the information output and the expected output was set. The nonlinear function weight of the overlapping area information of 3D medical images was modified by using the learning rate and the use time of the overlapping area information, and the influence factors of the overlapping information detection were obtained by increasing the situation terms, so as to complete the detection of the surface reconstruction information of 3D medical images. The experimental results show that the information detection results of the proposed method fit well with the actual situation, and the information detection efficiency is high.

Motion correction model based on multitask spatial temporal graph convolutional network

This study develops an intelligent sports evaluation system based on deep learning, designed to accurately recognize and count physical exercise movements. It also investigates the impact of physical activity on college students’ mental health, with a focus on the mediating role of social support. The system consists of three core modules: posture recognition, movement counting, and psychological prediction. The posture recognition module employs the Spatial Temporal Graph Convolutional Network (ST-GCN) and its improved variants to analyze skeletal motion. The movement counting module evaluates the accuracy of counting various sports actions using a self-constructed dataset. The psychological prediction module examines how social support mediates the relationship between physical exercise and mental health through an experimental design. Results indicate that the ST-GCN model achieves 93.7% accuracy and a 93.5% F1-score in posture recognition, demonstrating an effective balance between performance and computational efficiency. In movement counting, both missed detection and false detection rates remain low, with recognition rates reaching 95.7% for standard actions and 87.3% for non-standard actions. The psychological prediction module shows that social support significantly mediates the effect of physical exercise on mental health, accounting for 47.4% of the total effect. Overall, the intelligent sports evaluation system performs strongly in both posture recognition and movement counting, providing precise support for physical education and training. The study also confirms the crucial mediating role of social support in the relationship between physical activity and mental health among college students. These findings offer a theoretical foundation for improving physical education and student health management in higher education and hold practical value for application in university physical education programs, athletic training, and mental health interventions.

The collaborative energy scheduling of source‐load‐energy storage has great potential to meet the active control requirements of power‐distribution networks. In this study, a federated deep reinforcement learning framework was developed to facilitate collaborative energy scheduling and maximize the total economic benefit in a distribution network. Then, considering the application of Markov decision processes for energy scheduling, a spatial temporal graph convolutional network transformer based power generation packaging model for renewable energy sources was presented, and a collaborative energy scheduling strategy based on a federated duelling deep Q‐network was designed. The simulation results indicate that the developed collaborative scheduling strategy can maximize the economic benefits of a power distribution network while ensuring data privacy.

Most of the existing gait emotion recognition methods use spatial convolution or temporal convolution separately. Without integrating these two dimensions, it is easy to loss valid information for emotion analysis. In order to solve this problem, we propose a supervised learning method using Spatial Temporal Graph Convolutional Networks (ST-GCN) to classify emotions into positive, neutral and negative patterns. In addition, we made a new gait dataset called EmoGait3d including 742 real gait skeleton sequences extracted from videos, also annotated with emotion labels. According to the experiment results, our method achieves the accuracy of 92.2% which outperforms current state-of-art algorithms for gait emotion recognition.

Enhanced Topology Representation Learning for Skeleton-Based Human Action Recognition

This paper presents a realtime and accurate method for 3D eye gaze tracking with a monocular RGB camera. Our key idea is to train a deep convolutional neural network(DCNN) that automatically extracts the iris and pupil pixels of each eye from input images. To achieve this goal, we combine the power of Unet\cite{ronneberger2015u-net:} and Squeezenet\cite{iandola2017squeezenet:} to train an efficient convolutional neural network for pixel classification. In addition, we track the 3D eye gaze state in the Maximum A Posteriori (MAP) framework, which sequentially searches for the most likely state of the 3D eye gaze at each frame. When eye blinking occurs, the eye gaze tracker can obtain an inaccurate result. We further extend the convolutional neural network for eye close detection in order to improve the robustness and accuracy of the eye gaze tracker. Our system runs in realtime on desktop PCs and smart phones. We have evaluated our system on live videos and Internet videos, and our results demonstrate that the system is robust and accurate for various genders, races, lighting conditions, poses, shapes and facial expressions. A comparison against Wang et al.[3] shows that our method advances the state of the art in 3D eye tracking using a single RGB camera.