Siamese Convolutional Neural Network and Few-Shot Learning for Embedded Gesture Recognition

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Aiming to provide feasible solutions for the realization of the robust and natural myoelectric control systems, a novel myoelectric control scheme supporting gesture recognition and muscle force estimation is proposed in this study. Eleven grasping gestures abstracted from daily life are selected as the target gesture set. The high-density surface electromyography (HD-sEMG) of the forearm flexor and the grasping force signal are collected simultaneously. The synchronous prediction of gesture category and instantaneous force is realized by the multi-task learning (MTL) technique. Especially, a post-processing algorithm based on threshold method is conducted to overcome the influence of force variation on the accuracy of gesture recognition. The experimental results show that the proposed post-processing method can decrease the classification error significantly. Specifically, the overall gesture classification error is reduced by 27 ~ 30 percent compared with not using the post-processing method; and 16 ~ 24 percent compared with using classical post-processing methods. The whole scheme can realize the synchronous gesture recognition and force estimation with 9.35 ± 11.48% gesture classification error and 0.1479 ± 0.0436 root-mean-square deviation force estimation accuracy. Meanwhile, it is feasible in different number of electrodes and well meets the real-time requirement of the EMG control system in response time delay (about 28.22 ~ 113.16ms on average). The proposed framework provides the possibility for myoelectric control supporting synchronous gesture recognition and force estimation, which can be extended and applied in the fields of myoelectric prosthesis and exoskeleton devices.

With the rapid advancement of computer vision and deep learning technologies, the accuracy and efficiency of real-time gesture recognition have significantly improved. This paper introduces a gesture-controlled robot system based on thermal imaging sensors. By replacing traditional physical button controls, this design significantly enhances the interactivity and operational convenience of human–machine interaction. First, a thermal imaging gesture dataset is collected using Python3.9. Compared to traditional RGB images, thermal imaging can better capture gesture details, especially in low-light conditions, thereby improving the robustness of gesture recognition. Subsequently, a neural network model is constructed and trained using Keras, and the model is then deployed to a microcontroller. This lightweight model design enables the gesture recognition system to operate on resource-constrained embedded devices, achieving real-time performance and high efficiency. In addition, using a standalone thermal sensor for gesture recognition avoids the complexity of multi-sensor fusion schemes, simplifies the system structure, reduces costs, and ensures real-time performance and stability. The final results demonstrate that the proposed design achieves a model test accuracy of 99.05%. In summary, through its gesture recognition capabilities—featuring high accuracy, low latency, non-contact interaction, and low-light adaptability—this design precisely meets the core demands for “convenient, safe, and natural interaction” in rehabilitation, smart homes, and elderly assistive devices, showcasing clear potential for practical scenario implementation.

Wi-Fi sensing technology based on deep learning has contributed many breakthroughs in gesture recognition tasks. However, most methods concentrate on single domain recognition with high computational complexity while rarely investigating cross-domain recognition with lightweight performance, which cannot meet the requirements of high recognition performance and low computational complexity in an actual gesture recognition system. Inspired by the few-shot learning methods, we propose WiGR, a Wi-Fi-based gesture recognition system. The key structure of WiGR is a lightweight few-shot learning network that introduces some lightweight blocks to achieve lower computational complexity. Moreover, the network can learn a transferable similarity evaluation ability from the training set and apply the learned knowledge to the new domain to address domain shift problems. In addition, we made a channel state information (CSI)-Domain Adaptation (CSIDA) data set that includes channel state information (CSI) traces with various domain factors (i.e., environment, users, and locations) and conducted extensive experiments on two data sets (CSIDA and SignFi). The evaluation results show that WiGR can reach 87.8–94.8% cross-domain accuracy, and the parameters and the calculations are reduced by more than 50%. Extensive experiments demonstrate that WiGR can achieve excellent recognition performance using only a few samples and is thus a lightweight and practical gesture recognition system compared with state-of-the-art methods.

Recently, Siamese convolution neural networks have achieved remarkable results in the field of target tracking because of their balanced accuracy and speed. At the same time, Siamese convolution neural networks can solve the problem that the deep neural network can’t be updated in time and training data is insufficient. We propose an object tracking algorithm based on SE-CNN and Siamese convolution neural network. SE-CNN is added to the feature extraction submodule of SiameseRPN convolution neural network to enhance the quality of spatial encodings. During inference, a novel distractor-aware objective module is introduced to perform incremental learning. Benefit from the SE-CNN and distractor-aware objective module, Our algorithm performs well in the terms of accuracy and robustness in VOT2015, VOT2016, VOT2017 and OTB 2015.

A gesture recognition system using stereo vision and arm model fitting

This article explores the application of YOLOv11 algorithm in gesture recognition field to evaluate its performance in human-computer interaction (HCI). By introducing the YOLOv8 model and corresponding dataset for training, we obtained the confusion matrix predicted by the model, which shows that the model can accurately recognize most gestures, although there are a few cases of misidentification. When the IoU threshold is 0.5, the average accuracy (mAP) of the model steadily improves with the progress of training, indicating that the overall performance of the model in gesture detection tasks has been enhanced. In addition, even under stricter evaluation conditions where the IoU threshold was increased from 0.5 to 0.95, mAP still showed an upward trend, although the growth rate was not as significant as mAP50, which still demonstrated the improvement in model performance. Through the detection of the test set images, we found that the YOLOv11 model can effectively recognize gestures and accurately interpret their meanings, demonstrating high accuracy. This study not only demonstrates the potential of YOLOv11 in gesture recognition tasks, but also provides a new technological path for the future development of HCI field. Overall, the YOLOv11 algorithm has demonstrated strong performance and accuracy in gesture recognition, providing a more natural and intuitive way for interaction between smart devices and humans.

In recent years, wearable sensor-based gesture recognition is proliferating in the field of healthcare. It could be used to enable remote control of medical devices, contactless navigation of X-ray display and Magnetic Resonance Imaging (MRI), and largely enhance patients' daily living capabilities. However, even though a few commercial or prototype devices are available for wearable gesture recognition, none of them provides a combination of (1) fully open API for various healthcare application development, (2) appropriate form factor for comfortable daily wear, and (3) affordable cost for large scale adoption. In addition, the existing gesture recognition algorithms are mainly designed for discrete gestures. Accurate recognition of continuous gestures is still a significant challenge, which prevents the wide usage of existing wearable gesture recognition technology. In this paper, we present Gemote, a smart wristband-based hardware/software platform for gesture recognition and remote control. Due to its affordability, small size, and comfortable profile, Gemote is an attractive option for mass consumption. Gemote provides full open API access for third party research and application development. In addition, it employs a novel continuous gesture segmentation and recognition algorithm, which accurately and automatically separates hand movements into segments, and merges adjacent segments if needed, so that each gesture only exists in one segment. Experiments with human subjects show that the recognition accuracy is 99.4% when users perform gestures discretely, and 94.6% when users perform gestures continuously.

Ultigesture: A wristband-based platform for continuous gesture control in healthcare

This study presents a multimodal gesture recognition system based on the fusion of visual and strain sensor data, aiming to overcome the limitations of traditional gesture recognition methods in complex environments. A flexible strain sensor made from a CNTs-Graphene-TPU composite material was developed to create a smart glove with high sensitivity and stability, capable of accurately capturing finger flexion signals. Meanwhile, by integrating deep learning algorithms, a hybrid gesture recognition framework was constructed that combines visual and strain sensor data. Experimental results show that the system maintains high recognition accuracy (96.31%) across varying lighting conditions, significantly improving gesture recognition accuracy and system robustness. The system shows promising applications, particularly for gesture interaction and sign language translation in challenging lighting environments.

This paper describes a miniature, wearable wireless hand- gesture recognition (HGR) system called HANDIO. It obtains its input from not only traditional inertial sensors but also a muscle tension sensor (MTS). The addition of MTS enables recognition of a much broader range of intuitive hand gestures, particularly those involving the wrist, that would otherwise be difficult to distinguish by traditional inertial-only HGRs. Among MTSs, we choose an optical MTS over the conventional surface electromyography(sEMG) for the small size, low power consumption, wearing comfort, and good detection rate. This novel miniaturized design enables the whole system to be easily patched on the wrist area or integrated into a wearable device such as a wristband or watch without extra wiring. Experimental results show that a total of 8 hand gestures involving the wrist can be recognized with a detection rate over 93%. The average power consumption of the optical sensor is only around 258μW. This versatile system can also be used to detect other joint activities such as the elbow and knee joint.

Deep learning (DL) has been used for electromyographic (EMG) signal recognition and achieved high accuracy for multiple classification tasks. However, implementation in resource-constrained prostheses and human-computer interaction devices remains challenging. To overcome these problems, this paper implemented a low-power system for EMG gesture and force level recognition using Zynq architecture. Firstly, a lightweight network model structure was proposed by Ultra-lightweight depth separable convolution (UL-DSC) and channel attention-global average pooling (CA-GAP) to reduce the computational complexity while maintaining accuracy. A wearable EMG acquisition device for real-time data acquisition was subsequently developed with size of 36mm×28mm×4mm. Finally, a highly parallelized dedicated hardware accelerator architecture was designed for inference computation. 18 gestures were tested, including force levels from 22 healthy subjects. The results indicate that the average accuracy rate was 94.92% for a model with 5.0k parameters and a size of 0.026MB. Specifically, the average recognition accuracy for static and force-level gestures was 98.47% and 89.92%, respectively. The proposed hardware accelerator architecture was deployed with 8-bit precision, a single-frame signal inference time of 41.9μs, a power consumption of 0.317W, and a data throughput of 78.6 GOP/s.

Gestures are a natural means of communication between humans, and also a natural modality for human-computer interaction. Automatic recognition of gestures using computer vision is an important task in many real-world applications, such as sign language recognition, computer games control, virtual reality, intelligent homes, and assistive environments. In order for a gesture recognition system to be robust and deployable in non-laboratory settings, the system needs to be able to operate in complex scenes, with complicated backgrounds and multiple moving and skin-colored objects. In this paper we propose an approach for improving gesture recognition performance in such complex environments. The key idea is to integrate a face detection module into the gesture recognition system, and use the face location and size to make gesture recognition invariant to scale and translation. Our experiments demonstrate the significant advantages of the proposed method over alternative computer vision methods for gesture recognition.

Gesture recognition plays an important role in human–computer interaction. However, most existing methods are complex and time-consuming, which limit the use of gesture recognition in real-time environments. In this paper, we propose a static gesture recognition system that combines depth information and skeleton data to classify gestures. Through feature fusion, hand digit gestures of 0–9 can be recognized accurately and efficiently. According to the experimental results, the proposed gesture recognition system is effective and robust, which is invariant to complex background, illumination changes, reversal, structural distortion, rotation, etc. We have tested the system both online and offline which proved that our system is satisfactory to real-time requirements, and therefore it can be applied to gesture recognition in real-world human–computer interaction systems.

Since hyperspectral images (HSI) captured by different sensors often contain different number of bands, but most of the convolutional neural networks (CNN) require a fixed-size input, the generalization capability of deep CNNs to use heterogeneous input to achieve better classification performance has become a research focus. For classification tasks with limited labeled samples, the training strategy of feeding CNNs with sample-pairs instead of single sample has proven to be an efficient approach. Following this strategy, we propose a Siamese CNN with three-dimensional (3D) adaptive spatial-spectral pyramid pooling (ASSP) layer, called ASSP-SCNN, that takes as input 3D sample-pair with varying size and can easily be transferred to another HSI dataset regardless of the number of spectral bands. The 3D ASSP layer can also extract different levels of 3D information to improve the classification performance of the equipped CNN. To evaluate the classification and generalization performance of ASSP-SCNN, our experiments consist of two parts: the experiments of ASSP-SCNN without pre-training and the experiments of ASSP-SCNN-based transfer learning framework. Experimental results on three HSI datasets demonstrate that both ASSP-SCNN without pre-training and transfer learning based on ASSP-SCNN achieve higher classification accuracies than several state-of-the-art CNN-based methods. Moreover, we also compare the performance of ASSP-SCNN on different transfer learning tasks, which further verifies that ASSP-SCNN has a strong generalization capability.