Convolutional and recurrent neural network for human activity recognition: Application on American sign language.

Recurrent neural networks (RNNs) are powerful and robust neural networks that capture time dynamics via cycles in the graph. Compared to Convolutional Neural Networks (CNNs), RNNs consider both the current and the historical inputs which leads them to the standard approach for practitioners to address machine learning tasks involving sequential data. Compared to the achievements in these sequential learning tasks, RNNs pale in presenting a promising performance on human action recognition. Do we miss something? In this paper, we will conduct a detailed investigation of the RNNs for the human action recognition task. Several alternative recurrent architectures are designed and compared. Even simply, we find that adding the supervision signal at the end of an RNN can provide us with a comparable if not better performance than the previous state-of-the-art. We further analyze the different combinations of modalities and the effect of the length of the sequence. Compared to CNN-based recognition architecture, RNNs focus more on the action itself. Going deeper, we also find that when modeling long and difficult actions, RNNs perform much better than CNNs. It means a complex video action dataset may be required.

In recent years, Deep Learning (DL) models, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM), have been widely used for Human Activities Recognition (HAR). They have achieved considerable performance improvements over classical Machine Learning (ML) approaches due to their excellent feature representation capabilities. Simple human activities are performed in sequential order with no overlaps or concurrent actions. The paucity of labeled training activity samples, high computational cost, and system resource requirements of deep learning architectures compared to lightweight models are some of the research challenges confronting the HAR community. We framed the lightweight DL-based CNNs, RNNs, and LSTM model for HAR to tackle these research challenges. The RNNs and LSTM have single layers with softmax activation functions, whereas the one-dimensional CNNs have two convolutional and single max-pooling layers. To evaluate the performance of our models, we used the publicly available benchmark WISDM experimental dataset, which includes the reading of six activities (walking, jogging, upstairs, downstairs, sitting, and standing) performed by 36 participants using a single accelerometer sensor. The experimental result illustrates the efficacy of our models in activity recognition, demonstrating that they attain higher accuracy while being computationally efficient.

The currently deployed prediction models for strawberry fresh produce (FP) are based on either conventional machine learning (ML) or on simple deep learning (DL) models that are mostly applied for yield prediction. In this paper, we propose more comprehensive DL models that are applied for the first time to predict strawberry yield. The strawberry price is predicted as well directly from weather input parameters and yield. The strawberry price prediction is achieved using compound DL models such as Convolutional Long Short-Term Memory Recurrent Neural Network (CNN-LSTM). It is found that by adding attention, the performance of the compound models usually improves. After utilizing an aggregated performance measure to find the best model, the Attention-CNN-LSTM model proved to be the best compared to the rest of the deployed conventional ML models as well as the compound and simple DL models. The aggregated measure shows that this model is capable of precisely predicting strawberry prices five weeks ahead while maintaining the lowest prediction error and the highest model correlation.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

In this work, we design a neural network named MCRNN for classifying signals with slight distinction. This model combines advantages of both Convolutional Neural Network and Recurrent Neural Network, allowing the network to distinguish between long signals with long time-frequency maps. We use an architecture involving both convolutional layers, for extracting high-level features and reduce the dimension of data from time-frequency maps, and recurrent layers, for aggregating long-term dependencies. We apply techniques like Adam optimizer, batch normalization, and softmax layers, obtaining highly competitive results of 99.9% accuracy even in a low SNR of -2dB. Moreover, we conduct extensive experiment through a range of noisy scenarios from -10dB to 30dB. The model can be extended to signal dataset with short time-frequency maps easily. We leverage open source dataset like DeepSig as verification.

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Contents: Preface. Part I: Reminiscences. L. Fant, Two Memorable Meals With Ursula and Ed. R.M. Battison, American Sign Language Linguistics 1970-1980: Memoir of a Renaissance. B. Bragg, A Fish Story for Ursula and Other Remembrances. Part II: Historical and Comparative Analyses of Sign Languages. J. Woodward, Sign Languages and Sign Language Families in Thailand and Vietnam. D. McKee, G. Kennedy, Lexical Comparisons of Signs From American, Australian, British, and New Zealand Sign Languages. H. Lane, R.C. Pillard, M. French, Origins of the American Deaf-World: Assimilating and Differentiating Societies and Their Relation to Genetic Patterning. Part III: Language in the Visual-Spatial Modality. E.L. Newport, T. Supalla, Sign Language Research at the Millennium. P. Siple, Attentional Resources and Working Memory: A New Framework for the Study of the Impact of Deafness on Cognition. M. Wilson, K. Emmorey, When Does Modality Matter? Evidence From ASL on the Nature of Working Memory. P.B. Braem, T. Bram, A Pilot Study of the Expressive Gestures Used by Classical Orchestra Conductors. N. Frishberg, An Interpreter Creates the Space. Part IV: Linguistic Analysis of Sign Languages. S.D. Fischer, More Than Just Handwaving: The Mutual Contributions of Sign Language and Linguistics. R.B. Wilbur, Phonological and Prosodic Layering of Nonmanuals in American Sign Language. R.C. Loew, C.T. Akamatsu, M. Lanaville, A Two-Handed Manual Alphabet in the United States. E. Pizzuto, V. Volterra, Iconicity and Transparency in Sign Languages: A Cross-Linguistic Cross-Cultural View. M. Kuntze, Codeswitching in ASL and Written English Language Contact. S.K. Liddell, Indicating Verbs and Pronouns: Pointing Away From Agreement. Part V: Language Acquisition. J.G. Bettger, Viewing Deaf Children in a New Way: Implications of Bellugi and Klima's Research for Education. R.P. Meier, Shared Motoric Factors in the Acquisition of Sign and Speech. A. Abrahamsen, Explorations of Enhanced Gestural Input to Children in the Bimodal Period. D. Lillo-Martin, Early and Late in Language Acquisition: Aspects of the Syntax and Acquisition of Wh-Questions in American Sign Language. J. Reilly, Bringing Affective Expression Into the Service of Language: Acquiring Perspective Marking in Narratives. C.A. Padden, V.L. Hanson, Search for the Missing Link: The Development of Skilled Reading in Deaf Children. L.A. Petitto, On the Biological Foundations of Human Language. Part VI: The Neural Organization of Sign Language. A.R. Damasio, H. Damasio, Language and the Brain. D.P. Corina, Some Observations Regarding Paraphasia in American Sign Language. H. Poizner, D. Brentari, M.E. Tyrone, J. Kegl, The Structure of Language as Motor Behavior: Clues From Signers With Parkinson's Disease. V.A. Fromkin, On the Uniqueness of Language.

People around the world with speech and hearing impairment use a media of communication known as ‘Sign language’. In recent times, Sign language is omnipresent. However, there exists a challenge for people who do not know sign language, to communicate with people who can communicate exclusively using sign language. This gap can be bridged by using technologies of recent times to recognize gestures and design intuitive systems with deep learning. The aim of this paper is to recognise American Sign Language gestures dynamically and create an intuitive system which provides sign language translation to text and speech of various languages. The system uses Convolutional neural network, natural language processing, language translation and text-to-speech algorithms. It is capable of recognizing hand gestures dynamically and predicting the corresponding letters to form a desired sentence accurately.

Deep learning methodologies have had a major impact on performance across a wide variety of machine learning tasks, and speech recognition is no exception. We describe a set of deep learning techniques that proved to be particularly successful in achieving performance gains in word error rate on a popular large vocabulary conversational speech recognition benchmark task (“Switchboard”). We found that the best performance is achieved by combining features from both recurrent and convolutional neural networks. We compare two recurrent architectures: partially unfolded nets with max-out activations and bidirectional long short-term memory nets. In addition, inspired by the success of convolutional networks for image classification, we designed a convolutional net with many convolutional layers and small kernels that create a receptive field with more nonlinearity and fewer parameters than standard configurations. When combined, these neural networks achieve a word error rate of 6.2% on this difficult task; this was the best reported rate at the time of this writing and is even more remarkable given that human performance itself is estimated to be 4% on this data.

This research introduces a system for translating American Sign Language (ASL) utilising Convolutional Neural Networks. (CNNs). The system under consideration has the capability to identify manual gestures executed by persons communicating in American Sign Language (ASL) and subsequently convert them into written language, thereby facilitating uninterrupted communication between the deaf and hearing populations. The CNN model was validated and trained using a set of data made up of 78,300 pictures of movements in the ASL Alphabet written in American Sign Language (ASL). In order to improve the performance of the model, the pre-processing of the data included a number of stages, such as converting the photos to grayscale, normalising the pixel values, and enhancing performance of the model. In order to ease classification, fully connected layers were added after a succession of pooling and convolutional layers in CNN's design. After 15 epochs of training, the model attained a validation accuracy of 99.85%. The findings of this research demonstrate the viability of employing Convolutional Neural Networks (CNNs) in the creation of precise and effective American Sign Language (ASL) translation systems, which can serve as a means of communication for people with auditory impairments.

Sentiment analysis (SA) detects people’s opinions from text engaging natural language processing (NLP) techniques. Recent research has shown that deep learning models, i.e., Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer-based provide promising results for recognizing sentiment. Nonetheless, CNN has the advantage of extracting high-level features by using convolutional and max-pooling layers; it cannot efficiently learn a sequence of correlations. At the same time, Bidirectional RNN uses two RNN directions to improve extracting long-term dependencies. However, it cannot extract local features in parallel, and Transformer-based like Bidirectional Encoder Representations from Transformers (BERT) are the computational resources needed to fine-tune, facing an overfitting problem on small datasets. This paper proposes a novel attention-based model that utilizes CNNs with LSTM (named ACL-SA). First, it applies a preprocessor to enhance the data quality and employ term frequency-inverse document frequency (TF-IDF) feature weighting and pre-trained Glove word embedding approaches to extract meaningful information from textual data. In addition, it utilizes CNN’s max-pooling to extract contextual features and reduce feature dimensionality. Moreover, it uses an integrated bidirectional LSTM to capture long-term dependencies. Furthermore, it applies the attention mechanism at the CNN’s output layer to emphasize each word’s attention level. To avoid overfitting, the Guasiannoise and GuasianDroupout are adopted as regularization. The model’s robustness is evaluated on four English standard datasets, i.e., Sentiment140, US-airline, Sentiment140-MV, SA4A with various performance matrices, and compared efficiency with existing baseline models and approaches. The experiment results show that the proposed method significantly outperforms the state-of-the-art models.

Speech is a powerful means of expressing thoughts, emotions, and perspectives. However, accurately determining the emotions conveyed through speech remains a challenging task. Existing manual methods for analyzing speech to recognize emotions are prone to errors, limiting our understanding and response to individuals’ emotional states. To address diverse accents, an automated system capable of real-time emotion prediction from human speech is needed. This paper introduces a speech emotion recognition (SER) system that leverages supervised learning techniques to tackle cross-accent diversity. Distinctively, the system extracts a comprehensive set of nine speech features—Zero Crossing Rate, Mel Spectrum, Pitch, Root Mean Square values, Mel Frequency Cepstral Coefficients, chroma-stft, and three spectral features (Centroid, Contrast, and Roll-off) for refined speech signal processing and recognition. Seven machine learning models are employed, encompassing Random Forest, Logistic Regression, Decision Tree, Support Vector Machines, Gaussian Naive Bayes, K-Nearest Neighbors, ensemble learning, and four individual, hybrid deep learning models including Long short-term memory (LSTM) and 1-Dimensional Convolutional Neural Network (1D-CNN) with stratified cross-validation. Audio samples from diverse English regions are combined to train the models. The performance evaluation results of conventional machine learning and deep learning models indicate that the Random Forest-based feature selection model achieves the highest accuracy of up to 76% among the conventional machine learning models. Simultaneously, the 1D-CNN model with stratified cross-validation reaches up to 99% accuracy. The proposed framework enhances the cross-accent emotion recognition accuracy up to 86.3%, 89.87%, 90.27%, and 84.96% by margins of 14.71%, 10.15%, 9.6%, and 16.52% respectively.

Human activity recognition involves classifying times series data, measured at inertial sensors such as accelerometers or gyroscopes, into one of pre-defined actions. Recently, convolutional neural network (CNN) has established itself as a powerful technique for human activity recognition, where convolution and pooling operations are applied along the temporal dimension of sensor signals. In most of existing work, 1D convolution operation is applied to individual univariate time series, while multi-sensors or multi-modality yield multivariate time series. 2D convolution and pooling operations are applied to multivariate time series, in order to capture local dependency along both temporal and spatial domains for uni-modal data, so that it achieves high performance with less number of parameters compared to 1D operation. However for multi-modal data existing CNNs with 2D operation handle different modalities in the same way, which cause interferences between characteristics from different modalities. In this paper, we present CNNs (CNN-pf and CNN-pff), especially CNN-pff, for multi-modal data. We employ both partial weight sharing and full weight sharing for our CNN models in such a way that modality-specific characteristics as well as common characteristics across modalities are learned from multi-modal (or multi-sensor) data and are eventually aggregated in upper layers. Experiments on benchmark datasets demonstrate the high performance of our CNN models, compared to state of the arts methods.

In this paper we propose a novel recurrent neural network architecture for video-based person re-identification. Given the video sequence of a person, features are extracted from each frame using a convolutional neural network that incorporates a recurrent final layer, which allows information to flow between time-steps. The features from all timesteps are then combined using temporal pooling to give an overall appearance feature for the complete sequence. The convolutional network, recurrent layer, and temporal pooling layer, are jointly trained to act as a feature extractor for video-based re-identification using a Siamese network architecture. Our approach makes use of colour and optical flow information in order to capture appearance and motion information which is useful for video re-identification. Experiments are conduced on the iLIDS-VID and PRID-2011 datasets to show that this approach outperforms existing methods of video-based re-identification.