GesturePod

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Visual impairment often becomes noticeable to others through the visibility of a white cane. This public sign of blindness is an imperative signal that people understand, interpret and act in accordance with; however, the white cane is more than an external iconic aid. This study describes, explains and reflects on movement, assemblages, relationality, companionship, interconnectedness and learning opportunities that resonated across multiple white cane users’ experiences. Biographical pieces of two visually impaired people and their close networks, literature based on authors’ lived experiences and testimonies of the first author who is also visually impaired are brought together into a coherent narrative, where the white cane is the protagonist. This narrative deconstructs the omnipresent metanarrative ‘all blind people always use a white cane’ into a complete evocation that acknowledges the complexity of daily living that materiality and blindness entail. We use the white cane to rethink blindness and open a multisensory and multidimensional understanding.

Gesture recognition (GR) is an emerging and wide-ranging area of research. GR is extensively applied in sign language, Immersive game technology, and other computer interfaces, among others. People with visual impairments face challenges in completing tasks, including navigating environments, using technologies, and engaging in social interactions. Additionally, people face challenges in balancing their individuality with the need for protection in their day-to-day work. It is likely to recognize the communication of visually challenged and deaf people by recording their speech, and in comparison, with recent datasets, hence establishing their objectives. The conventional machine learning (ML) model attempts to utilize handcrafted features, but often underperforms in real-time environments. Deep learning (DL) models have become a sensation amongst investigators recently, making conventional ML techniques comparatively old. Therefore, this study presents a new approach, Enhancing Gesture Recognition for the Visually Impaired using Deep Learning and an Improved Snake Optimization Algorithm (EGRVI-DLISOA), in an IoT environment. The EGRVI-DLISOA approach is an advanced GR system powered by DL in an IoT environment, designed to provide real-time interpretation of gestures to assist the visually impaired. Initially, the EGRVI-DLISOA technique utilizes the Sobel filter (SF) technique for the noise elimination process. For feature extraction, the SqueezeNet model is utilized due to its efficiency in capturing meaningful features from complex visual data. For an accurate GR process, the long short-term memory (LSTM) approach is implemented. To fine-tune the hyperparameter values of the LSTM classifier, the improved snake optimization algorithm (ISOA) is utilized. The experimentation of the EGRVI-DLISOA technique is investigated under the hand gestures dataset. The comparison study of the EGRVI-DLISOA technique revealed a superior accuracy value of 98.62% compared to existing models.

This article explores the application of YOLOv11 algorithm in gesture recognition field to evaluate its performance in human-computer interaction (HCI). By introducing the YOLOv8 model and corresponding dataset for training, we obtained the confusion matrix predicted by the model, which shows that the model can accurately recognize most gestures, although there are a few cases of misidentification. When the IoU threshold is 0.5, the average accuracy (mAP) of the model steadily improves with the progress of training, indicating that the overall performance of the model in gesture detection tasks has been enhanced. In addition, even under stricter evaluation conditions where the IoU threshold was increased from 0.5 to 0.95, mAP still showed an upward trend, although the growth rate was not as significant as mAP50, which still demonstrated the improvement in model performance. Through the detection of the test set images, we found that the YOLOv11 model can effectively recognize gestures and accurately interpret their meanings, demonstrating high accuracy. This study not only demonstrates the potential of YOLOv11 in gesture recognition tasks, but also provides a new technological path for the future development of HCI field. Overall, the YOLOv11 algorithm has demonstrated strong performance and accuracy in gesture recognition, providing a more natural and intuitive way for interaction between smart devices and humans.

Myoformer: sEMG missing signal recovery for gesture recognition based on multi-channel self-attention mechanism

For a long time, visually impaired person uses a white cane to guide their way when travel outside. The white cane has been useful for the blinds in improving their mobility but unfortunately the white cane has its limitation. One of the shortcomings of the white cane is that, it could only detect the obstacles that are within the contact ranges of the white cane. This problem sometimes could cause the blind person to be in trouble because of insufficient time to detect and warn new obstacles in front of the blind person. This research proposes a walking stick system that has two functions; to classify an obstacles height whether it is low or high and to detect a front hole. The ability to detect the height of an obstacle will help the visually impaired to either step over or avoid the obstacle. The ability to detect a hole should help the visually impaired to avoid it in time. The walking stick will use two ultrasonic sensors for the detection of obstacle height, and a laser sensor for the detection of hole. A controller will be used to monitor and analyze the data from the sensors and feedback to the user through a vibration sensor and buzzer. The algorithm to differentiate the height of obstacles is working well and it is able to differentiate high or low obstacles. The laser ranging sensor has successfully been tested for hole detection. Therefore, the walking stick with ultrasonic and laser sensors will help more visually impaired to move around much faster and feeling more safer due to improved warning system for their movement.

With the rapid advancement of computer vision and deep learning technologies, the accuracy and efficiency of real-time gesture recognition have significantly improved. This paper introduces a gesture-controlled robot system based on thermal imaging sensors. By replacing traditional physical button controls, this design significantly enhances the interactivity and operational convenience of human–machine interaction. First, a thermal imaging gesture dataset is collected using Python3.9. Compared to traditional RGB images, thermal imaging can better capture gesture details, especially in low-light conditions, thereby improving the robustness of gesture recognition. Subsequently, a neural network model is constructed and trained using Keras, and the model is then deployed to a microcontroller. This lightweight model design enables the gesture recognition system to operate on resource-constrained embedded devices, achieving real-time performance and high efficiency. In addition, using a standalone thermal sensor for gesture recognition avoids the complexity of multi-sensor fusion schemes, simplifies the system structure, reduces costs, and ensures real-time performance and stability. The final results demonstrate that the proposed design achieves a model test accuracy of 99.05%. In summary, through its gesture recognition capabilities—featuring high accuracy, low latency, non-contact interaction, and low-light adaptability—this design precisely meets the core demands for “convenient, safe, and natural interaction” in rehabilitation, smart homes, and elderly assistive devices, showcasing clear potential for practical scenario implementation.

Hand joints-based gesture recognition using a neural network provides excellent performance in hand gesture recognition. However, during the collection of sequential skeletal datasets, joints identification through hand pose estimations usually includes noise and even errors, which often diminish the accuracy of gesture recognition. To promote the availability of hand gesture recognition for such noisy datasets, this paper presents a nested interval unscented Kalman filter (UKF) with long short-term memory (NIUKF-LSTM) network to improve the accuracy of hand gesture recognition from noisy datasets. This nested interval method with the UKF changes the distribution of the sigma points based on two sampling intervals. By considering the information of previous frames, the nested interval method helps the NIUKF-LSTM network revise the noise in the sequential hand skeletal data and improve the recognition accuracy. The experimental results showing the removal of noisy skeletal data from the dynamic hand gesture dataset demonstrate the effectiveness of our NIUKF-LSTM network, which achieves better performance than do other state-of-the-art methods.

Obstacle avoidance is a major challenge during independent mobility for blind or visually impaired (BVI) people. Typically, BVI people can only perceive obstacles at a short distance (about 1 m, in case they are using the white cane), and some obstacles are hard to detect (e.g . , those elevated from the ground), or should not be hit by the white cane (e.g . , a standing person). A solution to these problems can be found in recent computer-vision techniques that can run on mobile and wearable devices to detect obstacles at a distance. However, in addition to detecting obstacles, it is also necessary to convey information about them in real time. This contribution presents WatchOut , a sonification technique for conveying real-time information about the main properties of an obstacle to a BVI person, who can then use this additional feedback to safely navigate in the environment. WatchOut was designed with a user-centered approach, involving four iterations of online listening tests with BVI participants in order to define, improve and evaluate the sonification technique, eventually obtaining an almost perfect recognition accuracy. WatchOut was also implemented and tested as a module of a mobile app that detects obstacles using state-of-the-art computer vision technology. Results show that the system is considered usable and can guide the users to avoid more than 85% of the obstacles.

Aiming at the problem that the accuracy of gesture recognition is not high due to the camera's shooting angle and depth information being affected by illumination, this paper proposes a gesture recognition method based on image processing and SIFT-SVM. First construct a gesture recognition data set, perform image processing such as denoising and segmentation on the collected pictures, then extract the geometric features of gestures through the Scale-invariant feature transform (SIFT) algorithm, and finally use the support vector machine (SVM) to classifies and recognizes different detected gestures. The simulation experiment results show that the accuracy of the method proposed in this paper reaches 99.89%, it has strong robustness, and the recognition speed is faster to meet the needs of general applications.

Aiming to provide feasible solutions for the realization of the robust and natural myoelectric control systems, a novel myoelectric control scheme supporting gesture recognition and muscle force estimation is proposed in this study. Eleven grasping gestures abstracted from daily life are selected as the target gesture set. The high-density surface electromyography (HD-sEMG) of the forearm flexor and the grasping force signal are collected simultaneously. The synchronous prediction of gesture category and instantaneous force is realized by the multi-task learning (MTL) technique. Especially, a post-processing algorithm based on threshold method is conducted to overcome the influence of force variation on the accuracy of gesture recognition. The experimental results show that the proposed post-processing method can decrease the classification error significantly. Specifically, the overall gesture classification error is reduced by 27 ~ 30 percent compared with not using the post-processing method; and 16 ~ 24 percent compared with using classical post-processing methods. The whole scheme can realize the synchronous gesture recognition and force estimation with 9.35 ± 11.48% gesture classification error and 0.1479 ± 0.0436 root-mean-square deviation force estimation accuracy. Meanwhile, it is feasible in different number of electrodes and well meets the real-time requirement of the EMG control system in response time delay (about 28.22 ~ 113.16ms on average). The proposed framework provides the possibility for myoelectric control supporting synchronous gesture recognition and force estimation, which can be extended and applied in the fields of myoelectric prosthesis and exoskeleton devices.

This paper presents the development of a real time American Sign Language (ASL) detection system using Raspberry Pi and a Sequential Convolutional Neural Network (CNN) model. The system aims to bridge the communication gap between the Deaf and Hard of Hearing (DHH) community and the hearing population by translating ASL gestures into text and audio outputs. The proposed system leverages the Raspberry Pi 5, a cost effective and scalable hardware platform, combined with deep learning techniques to achieve high accuracy in gesture recognition. The system utilizes the custom dataset for training and employs hand landmark detection using MediaPipe for real-time gesture analysis. The results demonstrate an 85% accuracy in recognizing ASL gestures, with real-time text and audio outputs. The system is designed for personal, educational, and public applications, offering a practical solution for enhancing communication accessibility for the DHH community. Key Words: American Sign Language(ASL), Raspberry Pi, Machine Learning, Gesture recognition, Sign Language Translation, Text-to-Speech conversion

The visually-impaired people have the difficulty on perceive the largeness of a space and objects existed in a space by means of visual information. Particularly auditory-trained visually-impaired people can recognize 3-D spatial information with environmental sounds. However, systematic learning method of auditory training for acquired visually-impaired is not established sufficiently because the self-experience of the visually-impaired people is the main reason of ability acquisition in the actual environment because they do not have enough information about moving in the real environment. In this paper, the authors aim at demonstrating moving situations and moving needs of the visually-impaired, for example, which acoustical factors can be used in some living situation, what kind of environmental situation they think it difficult to perceive silent objects and what kind of aid they need. Results indicated following facts: Totally visually-impaired people tend to get more spatial information from auditory than not-totally visually-impaired people. Regarding available auditory cues, items of rotating head in order to listen carefully to environmental sound, and hitting floors stronger by a white cane or foot for voluming up reflected or reverberated sounds were much selected by the totally visually-impaired participants who can perceive obstacles with auditory information. Result of question about conventional devices indicated that most totally visually-impaired participants selected electrical sounding device informing obstacle distance, while most not-totally visually-impaired participants selected vibrating device informing the distance of walls or doors and tactually-stimulated device informing obstacle distance.

As an intuitive form of interaction, gestures are widely used in fields like smart live streaming and smart homes, significantly improving user efficiency and the convenience of human‐computer interaction. However, existing gesture recognition methods often suffer from large model sizes and high computational complexity, making them unsuitable for real‐time deployment. To address these issues, we propose GF‐YOLOv8, a lightweight gesture recognition model based on YOLOv8 (where ‘GF’ stands for the initials of two optimization modules: G represents the GB‐C2f module, and F represents the FFPN network), designed to reduce parameters while maintaining detection accuracy. First, by designing the GB‐C2f module, which integrates the lightweight network GhostNet, the model significantly reduces computation and parameters with minimal loss in accuracy. Second, a fusion feature pyramid network (FFPN), based on ASFF and AFPN algorithms, and enhances the interaction between non‐adjacent layers to improve gesture feature perception. Compared to the original YOLOv8 model, the proposed method demonstrates significant improvements on both public and self‐constructed datasets: GFLOPs and parameter count are reduced by 46.9% and 44.9%, respectively, with the mAP value remaining almost unchanged. The improved model enables efficient and accurate gesture recognition on mobile and embedded devices, achieving a substantial reduction in model size without compromising accuracy. © 2025 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

There are plenty of objects that are conspicuous in the way they signal to the surroundings that the user has some kind of physical impairment. Most obvious are perhaps wheelchairs, motorized wheelchairs, hearing aids and crutches. These objects can be described as aids in the sense that the purpose is either to mitigate the effects of reduced physical abilities or to compensate for the loss of a sensory faculty. The focus of this article concerns an object that both increases mobility and replaces such a faltering capacity. It might even be the most iconic of all aids: the mobility cane, also known as the white cane, long cane or white stick. Based on the work at a Swedish Low Vision and Resource Centre, this article discusses the use of the white cane among people with severe and progressive visual impairment. The fact is that this aid, which for an outsider would easily seem to be a rather natural choice, can arouse completely different feelings in somebody who has a severe visual impairment. It is a well-known circumstance amongst low visions teachers and therapists that persons with acquired or progressive sight impairment are often not quite happy about using the aid. Therefore, the aim of the article is to utilize notions of everyday experiences to gain further insights into why the white cane for many of its potential users are associated with strong feelings of both personal ambiguity and social stress. In order to highlight these emotional, and still cultural and political, tensions, the discussion is grounded in critical disability studies and informed by ableism, stigma, and passing as theoretical concepts.