Multiple-Weight Recurrent Neural Networks

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Speaker identification is the process of automatically determining who is speaking from the known speakers by the model. It is crucial in voice-based authentication, forensic investigations, security and surveillance. In recent studies, the combination of convolutional neural network (CNN) and recurrent neural network (RNN) variants performed better than separate models of both. However, only limited studies are conducted in speaker identification using a combination of CNN and RNN variants. In this study, we proposed speaker identification using hybrid two-dimensional CNN (2DCNN) and bidirectional gated recurrent unit (BiGRU) to improve performance. The proposed model integrates the advantage of 2DCNN and BiGRU layers to improve the performance of the model. 2DCNN layers have the advantage of extracting short-term spatial features from input data and it has a limited number of parameters for computation. BiGRU layers have an advantage in extracting long-term temporal dependency between the features in both directions (i.e. backward and forward) and it is efficient in achieving convergence during training. Spectrograms of the speech were used as input in our proposed model because of the rich acoustic features of the speaker. To compare the performance of the proposed model, additional experiments are conducted using the models 2DCNN, CNN-LSTM, CNN-BiLSTM and CNN-GRU. The experiments were conducted on the VoxCeleb1 audio dataset, which consists of 153,516 utterances collected from the 1251 speakers. The accuracy, precision, recall and f1 score of the proposed model are 98.28%, 99.08%, 98.92% and 98.97% respectively. The proposed model was compared with the existing works to show the effectiveness of the proposed model. The experiment results and the comparison with the existing works show that the proposed model has higher performance than both existing works and other models experimented in this study.

Recurrent neural networks (RNNs) can be used to handle sequential patterns and have been used for speech recognition. To overcome the shortcomings of RNN, recurrent sub neural networks (RSNNs) are used, where an RSNN is built independently for each class. The training algorithm of the RSNN is based on the backpropagation algorithm. Speaker dependent connected Chinese digit-speech recognition experiments were carried out. Some factors influencing the performance of RSNNs have been studied. The experiments show that RSNN is easier to train and gives higher performance than RNN.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Missing values are a common problem of time series such as sensor-generated data due to issues of data collection, transmission, and storage. To deal with this problem, various data imputation methods have been proposed. Among these methods, variants of recurrent neural networks (RNNs) have attracted a lot of attention as they provide a natural representation of temporal dynamics. Though such methods have been shown to achieve higher imputation accuracy, they mostly focus on the modeling of temporal dynamics from a short-term perspective while ignoring the long-term features such as trend and seasonal information embedded in time series data. In this paper, we investigate different paradigms of time series imputation methods to capture temporal dynamics from both long-term and short-term perspectives. In particular, we propose a new learning paradigm called MuSDRI which combines the time series decomposition method Seasonal Trend decomposition using Loess (STL) with RNNs. Specifically, STL is used to decompose long sequences of time series into trend, seasonal, and remainder components such that long-term dynamics can be revealed; RNNs are used to capture local patterns from the remainder component extracted by STL such that short-term dynamics can be refined. Moreover, MuSDRI is able to handle time series with multiple seasonal patterns by learning their contributions with respect to the imputation task. We evaluate the proposed imputation methods on three real-world datasets. Experimental results show that the imputation methods considering both the long-term and short-term dynamics achieve higher imputation accuracy, and among them MuSDRI in general achieves the best performance.

Artificial Intelligence (AI) has recently emerged as a powerful tool with versatile applications spanning various domains. AI replicates human intelligence processes through machinery and computer systems, finding utility in expert systems, image and speech recognition, machine vision, and natural language processing (NLP). One notable area with limited exploration pertains to using deep learning models, specifically Recurrent Neural Networks (RNNs), for predicting water quality in wastewater treatment plants (WWTPs). RNNs are purpose-built for handling sequential data, featuring a feedback mechanism. However, standard RNNs may exhibit limitations in accommodating both short-term and long-term dependencies when addressing intricate time series problems. The solution to this challenge lies in adopting Long Short-Term Memory (LSTM) cells, known for their inherent memory management through a ‘forget gate’ mechanism. In general, LSTM architecture demonstrates superior performance. WWTP data represent a historical series influenced by fluctuating environmental conditions. This study employs simple RNNs and LSTM architecture to construct prediction models for effluent parameters, systematically assessing their performance through various training data scenarios and model architectures. The primary objective was to determine the most suitable WWTP dataset model. The study revealed that an epoch setting of 50 and a batch size of 100 yielded the lowest training time and root mean square error (RMSE) values for both RNN and LSTM models. Furthermore, when these models are applied to predict effluent parameters, they exhibit precise RMSE values for all parameters. The study results can be applied to detect potential upsets in WWTP operations.

Recent advancement in artificial intelligence (AI) has resulted in the emergence of Machine Learning as a Service (MLaaS) as a lucrative business model which utilizes deep neural networks (DNNs) to generate revenue. With the investment of huge amount of time, resources, and budgets into researching and developing successful DNN models, it is important for us to protect its intellectual property rights (IPRs) as these models can be easily replicated, shared, or redistributed without the consent of the legitimate owners. So far, a robust protection scheme designed for recurrent neural networks (RNNs) does not exist yet. Thus, this chapter proposes a complete protection framework that includes both white-box and black-box protection to enforce IPR on different variants of RNN. Within the framework, a key gate was introduced for the idea of embedding keys to protect IPR. It designates methods to train RNN models in a specific way such that when an invalid or forged key is presented, the performance of the embedded RNN models will be deteriorated. Having said that, the key gate was inspired by the nature of RNN model, to govern the flow of hidden state and designed in such a way that no additional weight parameters were introduced.

Abstract: This chapter explores the cutting-edge advancements in computational language technology, delving into the transformative impact of recent innovations in the field. It covers the evolution from traditional rule-based and statistical methods to advanced neural network architectures, such as Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers. The chapter highlights key applications in natural language processing (NLP), including speech recognition, machine translation, and sentiment analysis, while addressing the ethical and societal implications of these technologies. Emphasis is placed on emerging research areas, such as multilingual NLP, low-resource language processing, and conversational AI. The chapter concludes by discussing future trends and challenges, including scalability, interpretability, and fairness in NLP systems, providing a comprehensive overview of the current state and future directions of computational linguistics. Keywords: Computational Linguistics,Natural Language Processing (NLP),Neural Networks,Deep Learning,Speech Recognition,Machine Translation,Sentiment Analysis,Multilingual NLP,Low-Resource Languages,Conversational AI,Ethical AI,Data Privacy,Algorithmic Bias and Future Trends in NLP.

With the increase in computing power and the availability of large amounts of data, deep learning techniques, especially convolutional neural networks (CNNS) and recurrent neural networks (RNNS), have become important tools for processing complex signals. These methods show excellent performance in speech recognition, image processing, natural language processing and so on. In this paper, we explore the application of recurrent neural network (RNN) sequence algorithms in the field of computer digital signal processing, highlighting current artificial intelligence techniques and their capabilities in solving complex signal processing problems. First, the paper reviews the basic principles and development of deep learning and RNN sequence algorithms, highlighting the advances these advanced technologies have made in simulating the way the human brain processes information. The practical application and effect of RNN sequence algorithm in computer digital signal processing are demonstrated through experimental data. By comparing with traditional algorithms, we demonstrate the efficiency and accuracy of RNN in processing complex signals, such as speech recognition in noisy environments and real-time video data processing. The experimental data not only demonstrate the effectiveness of RNNS in this field, but also highlight the unique advantages of deep learning methods when dealing with large and high-dimensional data. Through these empirical studies, this paper aims to provide researchers and engineers with an in-depth understanding of the potential of RNNS in digital signal processing applications, and looks forward to the future development direction of artificial intelligence technology in this field.

Internet of Things (IoT) devices are becoming more prevalent in a variety of businesses and for a variety of reasons (for example, sensing/collecting environmental data in both civilian and military situations). Because of their growing impact on many different uses and their expanding computational and analytical capacities, they are a potential threat victim for malware intended to hack particular IoT gadgets. For that, in this paper, we have proposed a successful Recurrent Neural Network (RNN) for malware detection. RNN is a sort of artificial neural network in which nodes are linked together to create a directed chart with a time sequence. Multiple trials with varied hyperparameter values in this research were trained. Our thorough trials demonstrated when employing RNN for malware categorization, the embedding size is more important than the input size. RNN performance with two different feature vectors was assessed using hyperparameters to validate RNN as an efficient solution for malware detection. Natural Language Processing (NLP) and feature selection are the two feature vectors. A paired t-test was also employed in this paper to see if the findings were meaningful to one another. In comparison to the chi2 feature selection, RNN with NLP attained the maximum AUC value and a reasonable variance. Based on the actual results. The proposed effective malware detection approach proves 99% detection accuracy with NLP techniques and 89% with feature selection techniques, hitherto results with the RNN classifier.

Parsing is a key task in computer science, with applications in compilers, natural language processing, syntactic pattern matching, and formal language theory. With the recent development of deep learning techniques, several artificial intelligence applications, especially in natural language processing, have combined traditional parsing methods with neural networks to drive the search in the parsing space, resulting in hybrid architectures using both symbolic and distributed representations. In this article, we show that existing symbolic parsing algorithms for context-free languages can cross the border and be entirely formulated over distributed representations. To this end, we introduce a version of the traditional Cocke–Younger–Kasami (CYK) algorithm, called distributed (D)-CYK, which is entirely defined over distributed representations. D-CYK uses matrix multiplication on real number matrices of a size independent of the length of the input string. These operations are compatible with recurrent neural networks. Preliminary experiments show that D-CYK approximates the original CYK algorithm. By showing that CYK can be entirely performed on distributed representations, we open the way to the definition of recurrent layer neural networks that can process general context-free languages.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Sequence modeling is a fundamental problem in Natural Language Processing (NLP) due to the sequential nature of language. With advances in Recurrent Neural Networks, this family of models has become the state-of-the-art for many sequence mapping tasks. In this thesis, we examine the effectiveness of this family of models for several classes of problems in NLP.

Recurrent transform learning

In this paper, a novel approach for implementing Arabic isolated speech recognition is described. While most of the literature on speech recognition (SR) is based on hidden Markov models (HMM), the present system is implemented by modular recurrent Elman neural networks (MRENN). The promising results obtained through this design show that this new neural networks approach can compete with the traditional HMM-based speech recognition approaches.

The natural language processing (NLP) area has been magically transformed by the Deep Neural Network (DNN). The two variations of Neural Networks, Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) can handle the different NLP tasks effectively. CNN is based on feature extraction using N-grams at higher levels. The RNN can be effectively used to model the sequential information. Choosing a neural technique for the various NLP tasks is always a challenge. The paper focuses on the evaluating the performance of the two DNN techniques CNN and RNN for the various NLP tasks, so that appropriate technique can be selected.