Gesture-Driven Cursor Control Interactive Projection System

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

This article explores the application of YOLOv11 algorithm in gesture recognition field to evaluate its performance in human-computer interaction (HCI). By introducing the YOLOv8 model and corresponding dataset for training, we obtained the confusion matrix predicted by the model, which shows that the model can accurately recognize most gestures, although there are a few cases of misidentification. When the IoU threshold is 0.5, the average accuracy (mAP) of the model steadily improves with the progress of training, indicating that the overall performance of the model in gesture detection tasks has been enhanced. In addition, even under stricter evaluation conditions where the IoU threshold was increased from 0.5 to 0.95, mAP still showed an upward trend, although the growth rate was not as significant as mAP50, which still demonstrated the improvement in model performance. Through the detection of the test set images, we found that the YOLOv11 model can effectively recognize gestures and accurately interpret their meanings, demonstrating high accuracy. This study not only demonstrates the potential of YOLOv11 in gesture recognition tasks, but also provides a new technological path for the future development of HCI field. Overall, the YOLOv11 algorithm has demonstrated strong performance and accuracy in gesture recognition, providing a more natural and intuitive way for interaction between smart devices and humans.

The way humans interact with material objects around them has evolved a lot over the ages, with new technologies being added to the lot over the years. Same goes for the human computer interaction, the basic mode comprising of mouse and various buttons and keyboards etc. has become obsolete now. New emerging technologies and ideas from various companies are setting a benchmark for the way this interaction can take place, with utmost ease and precision. From visual based guidance systems to recognition based on retina scan, fingerprints or gesture recognition, the possibilities are virtually endless. Gone are the days of needing a GUI for interacting with a computer application, each replaced by motion tracking softwares and firmwares for the same. This paper deals with three emerging technologies for the same, LEAP motion controller, Microsoft Kinect and Data Glove. Gesture recognition using data glove is an efficient choice for HCI but it has poor interface and it lags as far as computational time is concerned. Microsofts Xbox Kinect uses a high definition camera and an active IR feature to help recognize the body features and skeletal tracking to track movements of multiple users. Leap motion with its state of the art depth sensing camera and ability to render hand gestures using palm and fingers orientation in 3D space is yet another choice for HCI. A detailed description and working principles of each of the devices, along with the applications and advantages of one over the other are discussed in brief in this paper, along with the future prospects discussed at the end.

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

Hand joints-based gesture recognition using a neural network provides excellent performance in hand gesture recognition. However, during the collection of sequential skeletal datasets, joints identification through hand pose estimations usually includes noise and even errors, which often diminish the accuracy of gesture recognition. To promote the availability of hand gesture recognition for such noisy datasets, this paper presents a nested interval unscented Kalman filter (UKF) with long short-term memory (NIUKF-LSTM) network to improve the accuracy of hand gesture recognition from noisy datasets. This nested interval method with the UKF changes the distribution of the sigma points based on two sampling intervals. By considering the information of previous frames, the nested interval method helps the NIUKF-LSTM network revise the noise in the sequential hand skeletal data and improve the recognition accuracy. The experimental results showing the removal of noisy skeletal data from the dynamic hand gesture dataset demonstrate the effectiveness of our NIUKF-LSTM network, which achieves better performance than do other state-of-the-art methods.

Many researches and developments of table-top interface have been proposed in the last decade. In a table-top system, the difference of operations between the digital media and the physical media like a paper gives us unsatisfactory experience and disturbs a comfortable simultaneous using of them. It is required for user satisfaction that the user can operate the digital and physical media seamlessly. In this paper, we propose a table-top interface system that allows users to operate digital media intuitively by using gesture recognition. The proposed table can capture an image of the object on the table by press gesture recognition from users' nail images based on SVM, and can acquire a digital content from a digital paper reader by shake gesture of the device. The evaluation experiments show that the proposed system recognizes users' finger state with higher accuracy and the users can intuitively operate the table without awareness of the data transmission.

SUMMARYA variety of research on tabletop interfaces has been reported in the last decade. In a tabletop system, the difference in operation between digital media and physical media such as paper gives an unsatisfactory experience and disturbs their comfortable simultaneous use. For users to be satisfied, they must be able to operate the digital and physical media seamlessly. In this paper, we propose a tabletop interface system that allows users to operate digital media intuitively by using gesture recognition. The proposed table can capture an image of an object on the table by utilizing press gesture recognition from images of a user's fingernail based on support vector machine (SVM), and can acquire digital content from digital paper by utilizing a shake gesture of the device. The evaluation experiments show that the proposed system recognizes a user's finger state with high accuracy and that the users can intuitively operate the table without any awareness of data transmission.

The development of flexible pressure sensors for monitoring human motion and physiological signals has attracted extensive scientific research. However, achieving low monitoring limits, a wide detection range, large bending stresses, and excellent mechanical stability simultaneously remains a serious challenge. With the aim of developing a high-performance capacitive pressure sensor (CPS), this paper introduces the successful preparation of a single-walled carbon nanotube (SWNT)/polydimethylsiloxane (S-PDMS) composite dielectric with a foam-like structure (high permittivity and low elasticity modulus) and MXene/SWNT (S-MXene) composite film electrodes with a micro-crumpled structure. The above structurally modified CPS (SMCPS) demonstrated an excellent response output during pressure loading, achieving a wide pressure detection range (up to 700 kPa), a low detection limit (16.55 Pa), fast response/recovery characteristics (48/60 ms), enhanced sensitivity across a wide pressure range, long-term stability under repeated heavy loading and unloading (40 kPa, >2000 cycles), and reliable performance under various temperature and humidity conditions. The SMCPS demonstrated a precise and stable capacitive response in monitoring subtle physiological signals and detecting motion, owing to its unique electrode structure. The flexible device was integrated with an Internet of Things module to create a smart glove system that enables real-time tracking of dynamic gestures. This system demonstrates exceptional performance in gesture recognition and prediction with artificial intelligence analysis, highlighting the potential of the SMCPS in human-machine interface applications.

Non-verbal communication involves the usage of Sign Language. The sign language is used by people with hearing / speech disabilities to express their thoughts and feelings. But normally, people find it difficult to understand the hand gestures of the specially challenged people as they do not know the meaning of the sign language gestures. Usually, a translator is needed when a speech / hearing impaired person wants to communicate with an ordinary person and vice versa. In order to enable the specially challenged people to effectively communicate with the people around them, a system that translates the Indian Sign Language (ISL) hand gestures of numbers (1-9), English alphabets (A-Z) and a few English words to understandable text and vice versa has been proposed in this paper. This is done using image processing techniques and Machine Learning algorithms. Different neural network classifiers are developed, tested and validated for their performance in gesture recognition and the most efficient classifier is identified.

Gesture recognition by using Inertial Measurement Unit (IMU) sensors plays an important role in various Internet of Things (IOT) applications, e.g., smart home, intelligent medical system and so on. Traditional technologies usually utilize machine learning algorithms to train different gestures during the offline phase, then recognize the gesture during the online phase. However, such technologies cannot recognize these gestures without prior training. Even for the same gesture, with different gesture amplitude may result in unsuccessful recognition. Also if we change the person to perform the same gesture, the algorithms fails. In order to overcome these drawbacks, we propose an approach, which will be able to track the human body motion in realtime and also recognize complicated gestures. It utilizes the accelerometer information and proposes comprehensive localization algorithms for each deployed sensor attached on the human body. Then, it takes the correlation and limitation among body parts into account to recognize the gesture. Our experiments results show that, the successful recognition rate of our algorithm is 100%. Furthermore, any part of the human body can be well tracked, the tracking accuracy can reach 0.06m.

Recently, gesture recognition using visual sensors has been paid increased attention as an intelligent technology for human-computer interaction. To achieve good recognition performance in vision-based gesture recognition, it is important to find an efficient representation of complex visual signals and a robust classification method that can deal with diverse variations of gesture data. To treat these challenging topics, we propose a hierarchical structure for feature extraction and measuring similarity between two gestures obtained through RGB-D video sensors such as Kinect. The efficiency of the proposed method is confirmed through computational experiments on a public benchmark database.

This ppaper investigates the application of the YOLOv8 model in conjunction with Mediapipe and Squeeze-and-Excitation (SE) blocks for Indian Sign Language (ISL). ISL with its inherent complexities, presents unique challenges in gesture recognition. YOLOv8, a cutting-edge open-source classification model has shown promise in various image processing tasks, including segmentation and classification. However, its performance in ISL gesture recognition remains underexplored. In this research, the YOLOv8 model is integrated with Mediapipe, known for its efficient hand tracking and gesture recognition capabilities, to capture dynamic gesture features more accurately. Additionally, the incorporation of Squeeze-and-Excitation (SE) blocks enhances the model’s ability to focus on critical features by adaptively recalibrating channel-wise feature responses, thus improving classification accuracy. The correct placement and integration of SE block with YOLO architecture affects the features that the attention layer learns. This hybrid model architecture thoroughly evaluates various configurations and hyperparameters, offering valuable insights into the strengths and limitations of YOLOv8 combined with Mediapipe and SE blocks for ISL recognition. The results point out the importance of a diverse dataset and the limitations of improvement the addition of attention layer can bring to the current YOLO architecture used.

While myoelectric pattern recognition is a prevailing way for gesture recognition, the inherent nonstationarity of electromyography signals hinders its long-term application. This study aims to prove a hypothesis that morphological information of muscle contraction detected by ultrasound image is potentially suitable for long-term use. A set of ultrasound-based algorithms are proposed to realize robust hand gesture recognition over multiple days, with user training only at the first day. A markerless calibration algorithm is first presented to position the ultrasound probe during donning and doffing; an algorithm combining speeded-up robust features and bag-of-features model being immune to ultrasound probe shift and rotation is then introduced; a self-enhancing classification method is next adopted to update classification model automatically by incorporating useful knowledge from testing data; finally the performance of long-term hand gesture recognition with zero re-training is validated by a six-day experiment of six healthy subjects, whose outcomes strongly support the hypothesis with about 94% of gesture recognition accuracy for each testing day. This study confirms the feasibility of adoption of ultrasound sensing for long-term musculature related applications.

17(Suppl.): . Purpose: Most developed countries have been confronted with an acute growth in their elderly population. This situation is accompanied by increased prevalence of several health issues decreasing their quality of life. Alzheimer's disease is one of the most common cognitive disorders associated with aging. This has resulted in an urge to take advantage of technological tools to help doctors by providing early diagnosis which is vital for treatment of the elderly. These patients are not able to correctly emulate forelimb gestures and commit spatial/temporal errors, e.g. pretend to strike a nail with a hammer. 1 We developed a gesture recognition and evaluation tool to assist doctors in early detection of cognitive disorders (Figure 1). The framework performs precise gesture recognition based on state-of-the-art computer vision algorithms and evaluates subjects' performances to obtain a diagnostic report for clinicians. Method: The gesture recognition part of the framework uses four different methods. The first method employs local descriptors such as HOG/HOF 2 using shape and motion cues of a person performing a gesture. In the second method, skeleton information of the subjects is used to benefit from the body pose information. The third method uses depth information to detect exact hand patches with body part trajectories. We also trained a deep learning based recognition model 3 which empowered the framework to perform reliable recognitions. Results and Discussion: The framework is trained to recognize correct/incorrect performances. To have an objective decision on correctness of the gesture performances, clinicians annotated the data used for training the recognition algorithms. The utilized dataset was collected at the Institute Claude Pompidou (ICP) at Nice consisting of 14 dynamic and 15 static gestures selected by clinicians. The framework achieved 90% average accuracy on classifying gestures which is a reliable rate for diagnosis. Additionally, reaction and movement time of a subject are also detected. These two motor reflex criteria which measures responsive level of a subject are important factors in cognitive diagnosis. Motion descriptors are utilized to detect those quantities. All these types of information are illustrated in the user interface which provides clinicians with a detailed assessment of individual gesture and overall performance (Figure 1). Conclusion: We developed a user-friendly tool to help doctors in the diagnosis of cognitive disorders by providing a complete assessment of gestures performed by subjects. Future Work: We will conduct extensive evaluations on a larger population of patients with different levels of cognitive disorders. To improve gesture recognition task, more accurate deep models will be investigated.

This article discusses a method that performs gesture recognition, with the objective of extracting characteristics of the segmented hand, from dynamic images captured from a webcam and identifying signal patterns. With this method it is possible to manipulate simulated multirobots that perform specific movements. The method consists of the Continuously Adaptive Mean-SHIFT algorithm, followed by the Threshold segmentation algorithm and Deep Learning through Boltzmann restricted machines. As a result, an accuracy of 82.2%.