Dynamic gesture recognition based on feature fusion network and variant ConvLSTM

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

This paper deals with static and dynamic hand gesture (digits) recognition. The method provides a threefold novel contribution: (1) segmentation algorithm gives better results on any skin colour and any size of hand on complex and non-uniform background; (2) key frame finding algorithm and (3) the recognition technique of signs (static digits, alphabets and dynamic digits). We separate out key frames from a sequence of static gestures, which include correct gestures from a video sequence. The recognition efficiency of key frame detection is 93% using the proposed algorithm. The segmentation efficiency is almost 95%. Features are extracted using the proposed feature extraction algorithm, and gestures are recognised. We propose a novel algorithm for static and dynamic gesture recognition. The proposed algorithm shows recognition efficiency of 94.8% for static gestures and 94% for dynamic gestures.

Human-computer interaction (HCI) is an important feature of augmented reality (AR) technology. The naturalness is the inevitable trend of HCI. Gesture is the most natural and frequently used body auxiliary interaction mode in daily interactions except for language. However, there are often meaningless, subconscious gesture intervals between the two adjacent dynamic gestures. So, continuous dynamic gesture spotting is the premise and basis of dynamic gesture recognition, but there is no mature and unified algorithm to solve this problem. In order to realize the natural HCI based on gesture recognition entirely, a general AR application development platform is presented in this paper. According to the position and pose tracking data of the user's hand, the dynamic gesture spotting algorithm based on evidence theory is proposed. Firstly, Through analysis of the speed change of hand motion during the dynamic gestures, three knowledge rules are summed up. Then, accurate dynamic gesture spotting is realized with the application of evidence reasoning. Moreover, this algorithm first detects the starting point of gesture in the rising trend of hand motion speed, eliminates the delay between spotting and recognition, and thus ensures real-time performance. Finally, the algorithm is verified in several AR applications developed on the platform. There are two main experimental results. First, there are six users participating in the dynamic gesture spotting experiment, and the gesture spotting accuracy can meet the demand. Second, The accuracy of recognition after spotting is higher than that of the simultaneous recognition and spotting. So, It can be concluded that the proposed continuous dynamic gesture spotting algorithm based on Dempster-Shafer theory can extract almost all the effective dynamic gestures in the HCI of our AR platform, and on this basis, it can effectively improve the accuracy of the subsequent dynamic gesture recognition.

With the rapid development of human-computer interaction technology, gesture recognition becomes one of the key technologies of human-computer interaction. In this paper, we propose a new method of dynamic hand gestures recognition. The method adopts the hierarchical identification model for dynamic hand gestures recognition. First, we combine frame fusion with density distribution features for rough gesture recognition, second, we use the hausdorff distance or fingertip detection for accurate gesture recognition. The main innovation of this method lies in that we change the way of dynamic gestures recognition into the recognition of static image, improves the efficiency of gesture recognition effectively. Experimental results showed that our recognition rate is above 90%.

Dynamic gesture recognition systems face persistent challenges in achieving real-time performance and high recognition efficiency. This paper presents a novel framework integrating computer vision techniques with machine learning algorithms to address these issues. The approach utilizes OpenCV for dynamic gesture detection, employing gesture contour extraction using skin color feature segmentation, fingertip detection for static gesture recognition, and dynamic gesture tracking through Hidden Markov Models (HMM). Experimental results demonstrate high recognition accuracy of 95.8 ± 1.4% across various gesture types, with individual rates ranging from 95.2 ± 1.6% to 98.5 ± 0.8%. Real-time performance is achieved with average processing time of 82.71 ± 3.2 ms per frame (12.1 FPS). Our method demonstrates 1.15× to 1.49× speed improvements over state-of-the-art approaches while maintaining superior accuracy. Validation on the DHG-14/28 public benchmark confirms generalizability with 93.4 ± 1.6% accuracy. The framework shows robust performance under challenging conditions: low-light environments (92.4 ± 2.1%), fast motion (91.8 ± 1.9%), and complex backgrounds (93.2 ± 1.7%). Statistical significance was confirmed through comprehensive evaluation across diverse demographic groups. This research has implications for human-computer interaction applications, including smart home systems, augmented reality, and industrial automation.

Addressing the challenges of large-scale gesture recognition models, high computational complexity, and inefficient deployment on embedded devices, this study designs and implements a visual dynamic gesture recognition system based on a lightweight CNN-LSTM-DSA model. The system captures user hand images via a camera, extracts 21 keypoint 3D coordinates using MediaPipe, and employs a lightweight hybrid model to perform spatial and temporal feature modeling on keypoint sequences, achieving high-precision recognition of complex dynamic gestures. In static gesture recognition, the system determines the gesture state through joint angle calculation and a sliding window smoothing algorithm, ensuring smooth mapping of the servo motor angles and stability of the robotic hand's movements. In dynamic gesture recognition, the system models the key point time series based on the CNN-LSTM-DSA hybrid model, enabling accurate classification and reproduction of gesture actions. Experimental results show that the proposed system demonstrates good robustness under various lighting and background conditions, with a static gesture recognition accuracy of up to 96%, dynamic gesture recognition accuracy of 90.19%, and an overall response delay of less than 300 ms.

An action recognition network that combines multi-level spatiotemporal feature fusion with an attention mechanism is proposed as a solution to the issues of single spatiotemporal feature scale extraction, information redundancy, and insufficient extraction of frequency domain information in channels in 3D convolutional neural networks. Firstly, based on 3D CNN, this paper designs a new multilevel spatiotemporal feature fusion (MSF) structure, which is embedded in the network model, mainly through multilevel spatiotemporal feature separation, splicing and fusion, to achieve the fusion of spatial perceptual fields and short-medium-long time series information at different scales with reduced network parameters; In the second step, a multi-frequency channel and spatiotemporal attention module (FSAM) is introduced to assign different frequency features and spatiotemporal features in the channels are assigned corresponding weights to reduce the information redundancy of the feature maps. Finally, we embed the proposed method into the R3D model, which replaced the 2D convolutional filters in the 2D Resnet with 3D convolutional filters and conduct extensive experimental validation on the small and medium-sized dataset UCF101 and the large-sized dataset Kinetics-400. The findings revealed that our model increased the recognition accuracy on both datasets. Results on the UCF101 dataset, in particular, demonstrate that our model outperforms R3D in terms of a maximum recognition accuracy improvement of 7.2% while using 34.2% fewer parameters. The MSF and FSAM are migrated to another traditional 3D action recognition model named C3D for application testing. The test results based on UCF101 show that the recognition accuracy is improved by 8.9%, proving the strong generalization ability and universality of the method in this paper.

An Augmented Reality System for Coronary Artery Diagnosis Planning and Training (ARS-CADPT) is designed and realized in this paper. As the characteristic of ARS-CADPT, the algorithms of static gesture recognition and dynamic gesture spotting and recognition are presented to realize the real-time and friendly Human-Computer Interaction (HCI). The experimental results show that, with the use of ARS-CADPT, the HCI is natural and fluent, which improves the user’s immersion and improves the diagnosis and training effects.

Airwriting recognition is the task of identifying letters written in free space with finger movement. It is a dynamic gesture recognition with the vocabulary of gestures corresponding to letters in a given language. Electromyography (EMG) is a technique used to record electrical activity during muscle contraction and relaxation as a result of movement and is widely used for gesture recognition. Most of the current research in gesture recognition is focused on identifying static gestures. However, dynamic gestures are natural and user-friendly for being used as alternate input methods in human–computer interaction (HCI) applications. Airwriting recognition using EMG signals recorded from forearm muscles is, therefore, a viable solution. Since the user does not need to learn any new gestures and a large range of words can be formed by concatenating these letters, it is generalizable to a wider population. There has been limited work in recognition of airwriting using EMG signals and forms the core idea of the current work. The SurfMyoAiR dataset comprising of EMG signals recorded during writing English uppercase alphabets is constructed. Several different time-domain features to construct EMG envelope and two different time–frequency image representations: short-time Fourier transform and continuous wavelet transform were explored to form the input to a deep learning model for airwriting recognition. Several different deep learning architectures were exploited for this task. In addition, the effect of various parameters, such as signal length, window length, and interpolation techniques on the recognition performance, is comprehensively explored. The best-achieved accuracy was 78.50% and 62.19% in user-dependent and user-independent scenarios, respectively, by using short-time Fourier transform in conjunction with a 2-D convolutional neural network (CNN)-based classifier. Airwriting has great potential as a user-friendly modality to be used as an alternate input method in HCI applications.

A dynamic gesture recognition and understanding method in natural human-computer interaction under desktop environment is proposed, including the “reach”, “take up”, “move”, “put down”, “return”, “point” and other natural interactive gestures. In preprocess procedure of each frame of the video, the Gaussian background model and HSV skin-color model is employed to remove background and segment hand gestures. The temporal and spatial information of multi frame images is combined to construct temporal and spatial features of dynamic gestures images. Then a convolution neural network is built for recognize the dynamic characteristics of gesture image. Finally, the classification result is denoised to achieve the robust recognition and understanding of gestures. Experimental results show that the proposed method has a good ability of recognizing and understanding the dynamic gestures in the desktop environment.

International retailing: B. Sternquist, Fairchild Publications, New York, five parts, fifteen chapters, 592 pages, hardback, 1998, ISBN 1-56367-103-4

Low-pixel time-of-flight (ToF) sensors offer advantages in low power consumption, small size, and low cost. For gesture recognition, there are two advantages. Firstly, they are robust to changes in lighting conditions. Secondly, they could protect privacy. This paper addresses the challenge of utilizing an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{8}\times \mathbf{8}$</tex> pixel ToF sensor with extremely low resolution by proposing a signal processing system. The system includes pre-processing the data through non-global mean filtering to calibrate outliers. Additionally, we design a frames difference sliding window to ensure the integrity of gesture sequences and use Lagrangian interpolation to norm the fixed length. Finally, we design a tiny parallel convolutional network (TPC-Net) to recognize dynamic gestures. Experimental results demonstrate that the proposed system achieves an average accuracy of 98.47% over 6 dynamic gestures. Furthermore, our system achieves better performance with less model complexity compared to existing methods using low-pixel ToF sensors. These results illustrate the effectiveness of our system.

With the constant development of smart devices, Gesture recognition is applied on more and more fields. Gesture recognition devices currently on the market are inconvenient and expensive. A gesture recognition method based on Kinect is proposed in this paper. The camera of Kinect is used to get gesture images and then a hidden Markov model is established to recognize dynamic gesture so that the operation is simple. The assessment, decoding algorithms and training issues of hidden Markov model in gesture recognition is focused in this paper. Viterbi algorithm and Baum-Welch algorithm are used to solve the problems above. After simulation experiments, it is discovered that the gesture recognition algorithm based on Kinect proposed in this paper has a high accuracy rate for a variety of gestures, simultaneously the naturalness of human-computer interaction has been greatly improved.

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

In order to solve insufficient data features utilization efficiency and improve the performance of the deep neural network in image recognition, we propose a cross partial path weighted feature fusion (CPP) algorithm based on the Yolo framework. Firstly, CPP deepens the down-sampling depth of the backbone framework; then two weighted sampling partial paths are built in the feature fusion network, and the corresponding FPN layers are weighted and fused to form an output feature maps; finally, the output feature maps of different sizes are used to obtain more accurate bounding box position and category information. The experimental analysis results show that the advantage of our algorithm is to extend the network depth of the model as well as the network width. Our model be able to fuse more feature information so that it improves the recognition accuracy of object detection.