Control Virtual Human with Speech Recognition and Gesture Recognition Technology

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

Purpose To showcase how applications of automatic speech recognition (ASR) technology could help solve challenges in speech-language pathology practice with children with communication disability, and contribute to the realisation of the Sustainable Development Goals (SDGs). Result ASR technologies have been developed to address the need for equitable, efficient, and accurate assessment and diagnosis of communication disability in children by automating the transcription and analysis of speech and language samples and supporting dual-language assessment of bilingual children. ASR tools can automate the measurement of and help optimise intervention fidelity. ASR tools can also be used by children to engage in independent speech production practice without relying on feedback from speech-language pathologists (SLPs), thus bridging the long-standing gap between recommended and received intervention intensity. These innovative technologies and tools have been generated from interdisciplinary partnerships between SLPs, engineers, data scientists, and linguists. Conclusion To advance equitable, efficient, and effective speech-language pathology services for children with communication disability, SLPs would benefit from integrating ASR solutions into their clinical practice. Ongoing interdisciplinary research is needed to further advance ASR technologies to optimise children’s outcomes. This commentary paper focusses on industry, innovation and infrastructure (SDG 9) and partnerships for the goals (SDG 17). It also addresses SDG 1, SDG 3, SDG 4, SDG 8, SDG 10, SDG 11, and SDG 16.

In this paper, a brief overview derived out of detailed survey of speech recognition works reported in Indian languages is described. Robustness of speech recognition systems toward language variation is the recent trend of research in speech recognition technology. To develop a system which can communicate with human in any language like any other human is the foremost requirement in order to design appropriate speech recognition technology for one to all. India is a country which has vast linguistic variations among its billion plus population. Therefore, it provides a sound area of research toward language-specific speech recognition technology. From the beginning of the commercial availability of the speech recognition system, the technology has been dominated by the hidden Markov model (HMM) methodology due to its capability of modeling temporal structures of speech and encoding them as a sequence of spectral vectors. Most of the work done in Indian languages also uses HMM technology. However, from the last 10–15 years after the acceptance of neurocomputing as an alternative to HMM, artificial neural network (ANN)-based methodologies have started to receive attention for application in speech recognition. This is a trend worldwide as part of which few works have also been reported by a few researchers.

Gesture recognition technology is a natural human-computer interaction method with a wide range of applications, among which using bionic robotic hands to achieve gesture recognition is more meaningful and valuable for research. To improve the accuracy and the speed of gesture recognition, and also to facilitate the deployment of gesture recognition algorithm on edge device, a bionic robotic hand gesture recognition system is designed based on machine vision. Firstly, a bionic robot with six degrees of freedom based on Raspberry Pi 3B+ is chosen as the development platform of the system, including five finger degrees of freedom and one wrist degree of freedom; Secondly, a gesture recognition method is proposed based on a combination of feature extraction model, Inception v3 network, and migration learning, and it is trained and tested on a self-built gesture dataset. The results show that the method can accurately recognize gesture types with an average accuracy of 0.92; Finally, the trained model is deployed on a Raspberry Pi 3B+ bionic manipulator and tested on a physical system, and the average recognition speed is 0.17s after the system is stabilized, which shows that the system can effectively recognize and imitate hand gesture.

Speech recognition systems aim to make human-machine communication quickly and easily. In recent years, various researches and studies have been carried out to develop speech recognition systems. Examples of these studies are speech recognition, speaker recognition and speaker verification. In this study, speech recognition systems were investigated, methods used in the literature were investigated and a Turkish speech recognition application was developed. The application consists of speech coding and speech recognition. Firstly 20 Turkish words which are frequently used on the computer were determined. There are 20 records from each word. A total of 400 words were recorded on the computer with a microphone. In the speech coding section of the application, these words recorded on the computer are encoded by the Linear Pre-estimation Coding (LPC) method and the LPC parameters for each word are obtained. In the speech recognition section of the application, the Support Vector Machines (SVM) method is used. Two types of SVM classifiers are designed. These are the Soft Margin SVM (SM-SVM) classifier and the Least Square SVM (LS-SVM) classifier. Classification consists of training and testing stages. Of the 400 coded words, 200 were used for the training phase and 200 were used for the testing phase. As a result, 91% accurate recognition success for the SM-SVM classifier; 71% correct recognition of the LS-SVM classifier has been achieved.

Speech recognition technology has potential for application to intelligent transportation systems (ITS) in the area of advanced traveler information systems (ATIS). ATIS areas which can greatly benefit from speech recognition are: pre-trip traffic information; en-route traffic information; route guidance; ride matching and reservation; transit information services; and traffic and transit management control centers. This paper concentrates on telephony applications of speech recognition to two ATIS applications, real-time traffic information and public transit information. We first discuss the general considerations involved in development of a multiline speech interface to a traffic or transit database, including system architecture and sizing, and the development of dialogs and grammars which are user friendly and efficient. We then discuss two prototype applications of this technology using BBN's HARK speaker-independent speech recognition product. The first is to SmartRoute System's SmarTraveler traveler information system, which was an FHWA Operational Test, and the second to Montgomery County Maryland's public transit system. These were small scale tests to help us develop the technology required for full-scale implementations, but showed that the speech interface was practical, and provided much improved ease of use and performance over conventional touch tone based interactive voice response (IVR) systems.

Recently, deep neural network (DNN) with hidden Markov model (HMM) has turned out to be a superior sequence learning framework, based on which significant improvements were achieved in many application tasks, such as automatic speech recognition (ASR). However, the training of DNN-HMM requires the pre-segmented training data, which can be generated using Gaussian Mixture Model (GMM) in ASR tasks. Thus, questions are raised by many researchers: can we train the DNN-HMM without GMM seeding, and what does it suggest if the answer is yes? In this research, we come up with the ‘yes’ answer by presenting forward-backward learning algorithm for DNN-HMM framework. Besides, a training procedure is proposed, in which, the training for context independent (CI) DNN-HMM is treated as the pre-training for context dependent (CD) DNN-HMM. To evaluate the contribution of this work, experiments on ASR task with the benchmark corpus TIMIT are performed, and the results demonstrate the effectiveness of this research.

Gesture recognition is a way for computers to understand how humans move and express themselves without using traditional methods like typing or clicking. Instead of relying on text or graphics, gesture recognition focuses on reading body movements, such as those made by the hands or face. Currently, there is a specific interest in recognizing hand gestures by analyzing the veins on the back of the hand. Scientists have found that each person has a unique arrangement of veins beneath the skin of their hand. When the hand moves, the position of these veins changes, and this change is considered a gesture. These gestures are then translated into specific actions or tasks by coding the hand movements. This technology is particularly helpful for individuals with rotator cuff injuries. The rotator cuff is a group of muscles and tendons in the shoulder that can get injured, causing pain and limiting movement. People with these injuries may have difficulty steering a car, especially if their job or sport involves repetitive overhead motions. With gesture recognition technology, a person can control the car by simply moving their wrist, eliminating the need to use the shoulder. In summary, gesture recognition technology reads the unique patterns of hand veins to interpret hand movements, making it a practical solution for individuals with rotator cuff injuries who may struggle with certain tasks, like steering a car.

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

The study introduces a gesture recognition system for the classic "Rock, Paper, Scissors" game, based on a modified EfficientNetV2-B1 architecture. The dataset comprises 2,700 images, evenly divided among the three classes: "Rock", "Paper", and "Scissors". Leveraging the efficiency and accuracy of the EfficientNetV2-B1 model in image recognition tasks, the system was trained to classify these gestures effectively, and after fine-tuning, it achieved an accuracy of 98.89% and an Area Under the Curve (AUC) of ~1.0, indicating near-perfect classification across all classes. This performance highlights the potential of EfficientNetV2-B1 for real-time gesture recognition, with applications in interactive gaming and other gesture-based user interfaces. The proposed system also offers a foundation for further research and development in gesture recognition technologies.

In recent years, wearable sensor-based gesture recognition is proliferating in the field of healthcare. It could be used to enable remote control of medical devices, contactless navigation of X-ray display and Magnetic Resonance Imaging (MRI), and largely enhance patients' daily living capabilities. However, even though a few commercial or prototype devices are available for wearable gesture recognition, none of them provides a combination of (1) fully open API for various healthcare application development, (2) appropriate form factor for comfortable daily wear, and (3) affordable cost for large scale adoption. In addition, the existing gesture recognition algorithms are mainly designed for discrete gestures. Accurate recognition of continuous gestures is still a significant challenge, which prevents the wide usage of existing wearable gesture recognition technology. In this paper, we present Gemote, a smart wristband-based hardware/software platform for gesture recognition and remote control. Due to its affordability, small size, and comfortable profile, Gemote is an attractive option for mass consumption. Gemote provides full open API access for third party research and application development. In addition, it employs a novel continuous gesture segmentation and recognition algorithm, which accurately and automatically separates hand movements into segments, and merges adjacent segments if needed, so that each gesture only exists in one segment. Experiments with human subjects show that the recognition accuracy is 99.4% when users perform gestures discretely, and 94.6% when users perform gestures continuously.

Ultigesture: A wristband-based platform for continuous gesture control in healthcare

The demand for fast and accurate incremental speech recognition increases as the applications of automatic speech recognition (ASR) proliferate. Incremental speech recognizers output chunks of partially recognized words while the user is still talking. Partial results can be revised before the ASR finalizes its hypothesis, causing instability issues. We analyze the quality and stability of on-device streaming end-to-end (E2E) ASR models. We first introduce a novel set of metrics that quantify the instability at word and segment levels. We study the impact of several model training techniques that improve E2E model qualities but degrade model stability. We categorize the causes of instability and explore various solutions to mitigate them in a streaming E2E ASR system. Index Terms: ASR, stability, end-to-end, text normalization,on-device, RNN-T

Hand movements in our world are the most popular non-vocal ways of communication that are of great significance. Gesture recognition is an interaction with human computers, commonly used for purposes of education, medicine, and entertainment. Different computer vision related algorithms have used cameras for hand gesture recognition, but sturdy and robust detection of hand gestures is still difficult. We used the CNN model in this paper to define gesture and control devices using particular gestures. Gesture recognition technology will be easy, powerful and secure in this covid-19 pandemic situation.