Research on Gesture Recognition Method Based on Deep Learning

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

In this paper, a FPGA implementation method for convolutional neural networks of deep learning is proposed. In order to solve the problem of efficiency and speed of depth learning algorithms, a Z convolutional neural network structure is presented. The structure is easy to implement with FPGA. At the same time, in order to meet the requirement of complex multi-input and multi-output practical application system and increase the speed of calculations, a 3 dimensional Z type convolutional neural network structure is proposed, and the function of hardware circuit is verified by circuit simulation.

Artificial Neural networks have been proved most efficient in Deep Learning mainly because of large number of datasets it can handle. The most widely used is the Convolutional Neural Network (CNN). It has been proved useful for computer vision, pattern recognition and Natural Language Processing (NLP). CNN is so vastly used, as, unlike traditional Neural Nets, it reduces number of parameters and focus more on domain specific features. There are various CNN architectures proposed, such as LeNet, AlexNet, GoogleNet. In this paper, we talk about structure of CNN and all the models of CNN which are proposed till date.

Recently, deep learning has become a promising technique for constructing gesture recognition classifiers from surface electromyography (sEMG) signals in human–computer interaction. In this paper, we propose a gesture recognition method with sEMG signals based on a deep multi-parallel convolutional neural network (CNN), which solves the problem that traditional machine learning methods may lose too much useful information during feature extraction. CNNs provide an efficient way to constrain the complexity of feedforward neural networks by weight sharing and restriction to local connections. Sophisticated feature extraction is to be avoided and hand gestures are to be classified directly. A multi-parallel and multi-convolution layer convolution structure is proposed to classify hand gestures. Experiment results show that in comparison with five traditional machine learning methods, the proposed method could achieve higher accuracy.

Gesture recognition is popular in human-computer interaction research, and has broad application prospect in human disorder, sign language awareness, and other fields. Gesture recognition makes human-computer interaction operation more convenient. This paper proposes a gesture recognition system based on convolution neural network improved LeNet5 model. Based on raspberry pie hardware platform, the system captures and extracts gesture features by using OpenCV, and realizes gesture recognition by using Keras to train convolutional neural network model.

An attribution graph-based interpretable method for CNNs

Due to the irregular and sporadic nature of wildfires, continuous monitoring of large areas is required. Since geostationary satellite sensors can observe large areas with high temporal resolution, they are suitable for monitoring wildfires in real time. However, the threshold algorithm currently employed for satellite-based active fire detection has poor performance in sensors with low spatial resolution. In addition, the algorithm does not account for environmental conditions that affect wildfire detection, resulting in poor generalization performance for large areas. This study examines the viability of an adaptive active fire detection model by combining satellite and numerical model data with deep learning. A model for active fire detection was developed using commonly employed brightness temperature-related variables (key variables) and local environmental variables (sub variables). Key variables are the cross spectral and spatial differences between the MIR (central wavelength of 3.85 m) and 2 TIR (central wavelengths of 9.63 and 11.20 m) channels of the Advanced Himawari Imager (AHI). Sub variables include Solar zenith angle (SOZ) and satellite zenith angle (SAZ) of AHI, skin temperature (ST) and relative humidity (RH) of European Centre for Medium-Range Weather Forecasts (ECMWF) Reanalysis v5 (ERA5)-land data. Four processes (confidence, frequency, land cover, and continuity tests) were used to extract reference fire samples from Moderate Resolution Imaging Spectroradiometer (MODIS) and Visible Infrared Imaging Radiometer Suite (VIIRS) active fire products. To consider the different properties of key and sub variables, a 2-way convolutional neural network (CNN) structure was developed. To evaluate the influence of environmental variables, a CNN model without sub variables was adopted as a control model. The 2-way CNN (recall of 0.86, precision of 0.96, and standard deviation of recall of 0.13) was more robust at five focus sites than the control CNN (recall of 0.82, precision of 0.97, and standard deviation of recall of 0.163). Despite having a lower spatial resolution than MODIS/VIIRS, 2-way CNN outperformed other satellite-based active fire products (MODIS, VIIRS, AHI, and Advanced Meteorological Imager) in terms of detection capacity. The control CNN demonstrated poor performance under certain environmental conditions (high RH, high SAZ, and transition time between day and night), but 2-way CNN mitigates this tendency. In particular, the use of RH improved detection sensitivity, and SAZ contributed to the spatial robustness. This study demonstrated the significance of environmental conditions in active fire detection and proposed a suitable CNN structure for this intent. Based on the findings of this study, higher-level adaptive active fire monitoring under diverse environmental conditions will be possible together with explainable artificial intelligence.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Rapid quantitative analysis of cotton-polyester blended fabrics using near-infrared spectroscopy combined with CNN-LSTM

Gesture recognition, as an important means of human-computer interaction, can achieve more natural and flexible human-computer interaction, so it has been widely concerned by researchers in the field of computer vision. At present, most gesture recognition algorithms are based on monocular visual images and recognize the apparent features of hands. Most gesture image segmentation methods are carried out in color space according to skin color information. These methods are highly susceptible to interference from the external environment, such as lighting, background, etc. Convolutional neural network has the advantages of strong anti-interference and outstanding self-organization and self-learning ability. Therefore, based on the principle of convolutional neural network, a novel deep convolutional neural network dedicated to gesture recognition was designed in this paper. This network combines skin color information with finger position information for gesture recognition. Experimental results showed that the algorithm based on fingertip position information has better performance than the algorithm based solely on skin color information. Moreover, the network has simple structure and few parameters. Compared with VGG16 and other classical networks, the recognition accuracy is basically the same under the premise of fewer parameters and structural layers, and the recognition effect is better than other classical networks.

Human action recognition (HAR) has a considerable place in scientific studies. Additionally, hand gesture recognition, which is a subcategory of HAR, plays an important role in communicating with deaf people. Convolutional neural network (CNN) structures are frequently used to recognize human actions. In the study, hyperparameters of the CNN structures, which are based on AlexNet model, are optimized by heuristic optimization algorithms. The proposed method is tested on sign language digits and Thomas Moeslund’s gesture recognition datasets. Due to using heuristic algorithms, training procedures are repeated 30 times for both datasets. According to the experimental results, the average accuracy performance for action classification of the proposed artificial bee colony-based method is 98.40%, which is better than the performance of the existing work (with accuracy of 94.2%) for sign language digits dataset. Concurrently, for Thomas Moeslund’s gesture recognition dataset, the proposed approach has an average accuracy performance of 98.09%, outperforming the best existing work (which reported 94.33% classification performance).

Deep convolutional neural networks (DCNNs) for diagnosis of disease on chest radiographs (CXR) have been shown to be biased against males or females if the datasets used to train them have unbalanced sex representation. Prior work has suggested that DCNNs can predict sex on CXR, which could aid forensic evaluations, but also be a source of bias. To (1) evaluate the performance of DCNNs for predicting sex across different datasets and architectures and (2) evaluate visual biomarkers used by DCNNs to predict sex on CXRs. Chest radiographs were obtained from the Stanford CheXPert and NIH Chest XRay14 datasets which comprised of 224,316 and 112,120 CXRs, respectively. To control for dataset size and class imbalance, random undersampling was used to reduce each dataset to 97,560 images that were balanced for sex. Each dataset was randomly split into training (70%), validation (10%), and test (20%) sets. Four DCNN architectures pre-trained on ImageNet were used for transfer learning. DCNNs were externally validated using a test set from the opposing dataset. Performance was evaluated using area under the receiver operating characteristic curve (AUC). Class activation mapping (CAM) was used to generate heatmaps visualizing the regions contributing to the DCNN's prediction. On the internal test set, DCNNs achieved AUROCs ranging from 0.98 to 0.99. On external validation, the models reached peak cross-dataset performance of 0.94 for the VGG19-Stanford model and 0.95 for the InceptionV3-NIH model. Heatmaps highlighted similar regions of attention between model architectures and datasets, localizing to the mediastinal and upper rib regions, as well as to the lower chest/diaphragmatic regions. DCNNs trained on two large CXR datasets accurately predicted sex on internal and external test data with similar heatmap localizations across DCNN architectures and datasets. These findings support the notion that DCNNs can leverage imaging biomarkers to predict sex and potentially confound the accurate prediction of disease on CXRs and contribute to biased models. On the other hand, these DCNNs can be beneficial to emergency radiologists for forensic evaluations and identifying patient sex for patients whose identities are unknown, such as in acute trauma.

Aims: This study aimed to explore the effect of training set diversity on the performance of deep learning models for predicting early gastric cancer (EGC) under endoscopy. Methods: Images of EGC and non-cancerous lesions under narrow-band imaging (ME-NBI) and magnifying blue laser imaging (ME-BLI) were retrospectively collected. Training set 1 was composed of 150 non-cancerous and 309 EGC ME-NBI images, training set 2 was composed of 1505 non-cancerous and 309 EGC ME-BLI images, and training set 3 was the combination of training set 1 and 2. Test set 1 was composed of 376 non-cancerous and 1052 EGC ME-NBI images, test set 2 consisted of 529 non-cancerous and 71 EGC ME-BLI images, and test set 3 was the combination of test set 1 and test set 2. Three deep learning models, convolutional neural network (CNN) 1, CNN 2 and CNN 3 (CNN 1, CNN 2 and CNN 3 were independently trained using training set 1, training set 2 and training set 3, respectively), were constructed, and their performances on each test set were respectively evaluated. One hundred and thirty-eight ME-NBI videos and 17 ME-BLI videos were further collected to evaluate and compare the performance of each model in real time. Results: On the whole, the performance of CNN 3 was the best. The accuracy (Acc), sensitivity (Sn), specificity (Sp) and area under the curve (AUC) of test set 1 in CNN 3 were 87.89% (1255/1428), 90.96% (342/376), 86.79% (913/1052) and 94.60%, respectively. The Acc, Sn, Sp and AUC of test set 2 in CNN 3 were 95% (570/600), 97.92% (518/529), 73.24% (52/71) and 90.93% respectively. The Acc, Sn, Sp and AUC of test set 3 in CNN 3 were 89.99% (1825/2028), 95.03% (860/905), 85.93% (965/1123) and 94.89%, respectively. The performance of CNN 3 was also the best in videos test set. The Acc, Sn and Sp of videos test set in CNN 3 were 91.03% (142/156), 90.58% (125/138) and 94.44% (17/18), respectively. Conclusions: The deep learning model with the most diverse training data has the best diagnostic effect.

Convolutional neural networks (CNNs) have performed exceptionally well in various computer vision tasks. Previously, researchers relied on feature extraction and then classification. With CNNs, feature extraction and classification are performed in a single step. Therefore, incorporating a set of convolutional neural networks, known as an ensemble, can help increase the effectiveness of their behavior. This paper presents an architecture that is the result of collaborative efforts, consisting of three types of CNN: Densenet121, Resnet101, and MobileNetV2. We also present experimental results using the Cifar10 and Cifar100 datasets, achieving impressive classification accuracy of 99% for Cifar10 and 86% for Cifar100. This case study contributes to deep learning optimization and benefits researchers and practitioners looking for optimal approaches in various computer vision applications. We also evaluate the scalability and robustness of our proposed method in the context of different CNN structures by using more than one model.

This work aims to construct and optimize the public emotion network communication model under the artificial intelligence (AI) environment and provide a new research direction for exploring public emotion network communication with intelligent human–computer interaction (HCI). The main challenge of this research is the lack of interactivity of intelligent human–computer expression design, and the accuracy of emotion recognition is not high. First, the theoretical analysis of network public opinion is made, and the network public opinion calculation method and emotion recognition method based on deep learning are proposed. On this basis, the public emotion network communication model is established, and the model’s communication mechanism and rules are described in detail. Next, the emotion calculation of human–computer expression interaction and the emotion expression mode of the humanoid robot are proposed, which provides an experimental basis for HCI and public emotion network communication analysis. According to the data of email network and microblog network in the real network, the network degree distribution map is established. The statistical results show that email and microblog networks belong to power law distribution, and the aggregation coefficient is relatively high. Microblog network degree changes slowly, and more points are scattered in the middle area. In a practical sense, when the positive and negative emotions in the public emotion network communication model are input into the bionic robot, the robot can show happy and lost expressions. The results show that compared with the traditional convolutional neural network (CNN) model or recurrent neural network (RNN) model, the RNN–CNN structure model of RNN combined with CNN reduces the waiting time by about 20% and improves the algorithm accuracy by at least 3.1%. The public emotion network communication model based on deep learning and intelligent HCI can accurately reflect the public emotion state in the network, which provides a practical basis for the application of AI technology in network public opinion judgment.