A Review of Deep Learning with Recurrent Neural Network

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Short-term solar radiation forecasting using hybrid deep residual learning and gated LSTM recurrent network with differential covariance matrix adaptation evolution strategy

The dynamic modulus (|E*|) of hot-mix asphalt (HMA) is a crucial mechanistic characteristic essential in defining the strain response of asphalt concrete (AC) mixtures under varying loading rates and temperatures. This paper aims to conduct a comprehensive investigation of classical machine learning (ML) and deep learning (DL) algorithms as applied to the prediction of |E*| and compare their performance with renowned |E*| regression models (Witczak NCHRP 1-37A, Witczak NCHRP 1-40D, and Hirsch). Eight state-of-the-art ML and DL algorithms are attempted with diverse structures, including multiple linear regression (MLR), decision trees (DT), support vector regression (SVR), ensemble trees (ET), Gaussian process regression (GPR), artificial neural networks (ANN), recurrent neural networks (RNN), and convolutional neural networks (CNN). A comprehensive database was assembled, incorporating 50 AC mixtures, of which 25 were from the Kingdom of Saudi Arabia and 25 were from the state of Idaho, USA. This database encompasses an extensive dataset of 3,720 |E*| measurements, associated with thirteen input features representing the proposed AC mixtures’ aggregate gradations, binder characteristics, and volumetric properties. This pioneering study surpasses existing research by examining various algorithms to predict |E*| on the same dataset, applying them with different structures and individual optimization to achieve optimal performance. The developed models are evaluated based on multi-stage assessment criteria, including the accuracy and complexity performance measures and rationality based on a sensitivity analysis. The multi-stage comparative analysis results reveal that the bagging ETs, GPR with exponential kernel, and DT record the highest prediction accuracy; however, only the bagging ETs yield the highest accuracy, lowest training and testing complexity, and rational trends throughout the sensitivity analysis. The research outcome has the potential to provide pavement engineers with advanced tools for predicting |E*| and, therefore, optimizing pavement designs and rehabilitations. Doi: 10.28991/CEJ-2025-011-01-06 Full Text: PDF

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

Large scale artificial neural networks (ANNs) have been widely used in data processing applications. The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Such a unique architecture enables the recurrent neural network to remember the past processed information and makes it an expressive model for nonlinear sequence processing tasks. However, the large computation complexity makes it difficult to effectively train a recurrent neural network and therefore significantly limits the research on the recurrent neural network in the last 20 years. In recent years, the use of graphics processing units (GPUs) becomes a significant advance to speed up the training process of large scale neural networks by taking advantage of the massive parallelism capabilities of GPUs. In this paper, we propose an efficient GPU implementation of the large scale recurrent neural network and demonstrate the power of scaling up the recurrent neural network with GPUs. We first explore the potential parallelism of the recurrent neural network and propose a fine-grained two-stage pipeline implementation. Experiment results show that the proposed GPU implementation can achieve 2 ~ 11 x speed-up compared with the basic CPU implementation with the Intel Math Kernel Library. We then use the proposed GPU implementation to scale up the recurrent neural network and improve its performance. The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset.

Deep learning for time series sequence individual data instance classification can revolutionize computer assisted navigation by providing surgeons with accurate, real-time instrument locality through automatic instrument localization. This paper presents an evaluation of Deep Learning models to perform individual data instance classification of time series data. The models explored include convolution and recurrent networks, as well as state-of-the-art residual and inception architectures. The time series data used to evaluate the models consists of depth and force measurements from a drill. Four recurrent neural network models using long short-term memory and gated recurrent units, known as baseline models, and four models using 1D convolution with ResNet and Inception architectures, known as advanced models, were evaluated by determining the data instance membership of the four classes. The four classes represent four distinct regions in a bone traversed by the drill bit during a surgical procedure. First, the time series data is preprocessed, identifying the four classes or regions of the bone. Next, the paper presents a discussion of the network architecture and modifications of both the basic and advanced deep learning models, followed by the training process and hyperparameters tuning. The performance of the models was evaluated using the precision and recall performance parameters. Out of the eight models evaluated, the recurrent neural network with gated recurrent units has the best performance. The paper also demonstrates the importance of the feature depth over the feature force in classifying the data instances, followed by the effects of the imbalanced dataset on the performance of the models.

In recent years, successful applications of deep learning (DL) have inspired scientists to explore the possibilities of applying DL approaches to modeling scientific problems. Existing studies have revealed that to bake the physics into the DL makes a good supplement to the traditional numerical methods (e.g., finite element, finite volume method) which primarily rely on partial differential equations (PDEs). While DL models are ordinarily trained in a purely data-driven manner, integrating physics into them for simulating scientific problems has several benefits such as (i) physics constraints could regularize the over-parameterized model and hence mitigate the overfitting issue commonly seen in DL; (ii) physics information could also effectively reduce the amount of data needed for training the model; (iii) the resultant physics-informed DL models feature better interpretability and generalizability compared with the conventional black-box model. Furthermore, the powerful expressiveness of the deep network, guaranteed by the universal approximation theorem, makes it a suitable approximator for the solution to a physical system. In this dissertation, we develop two different DL architectures (or approaches), one being continuous scheme-based while the other discrete scheme-based, that leverage physics knowledge for modeling scientific problems. Through comprehensive numerical experiments, we demonstrate the proposed models can be used in solving general PDEs, establishing predictive data-driven models for dynamical systems, identifying the parameters in governing PDEs or even discovering the entire governing PDEs of dynamical systems from scarce and noisy measurements. The continuous model roots on the physics-informed neural network (PINN) which uses a fully connected neural network (FCNN) to approximate the physical fields of a system globally. This model is mesh-free as the residual of the physics (e.g., PDEs, initial/boundary values) is evaluated on a set of collocation points within the physical domain. Several applications including the forward simulations, data-driven simulations and solving inverse problems are presented to exemplify the advantages of PINN over traditional numerical methods. However, the original PINN suffers from inaccurate initial/boundary values due to the weak enforcement of the initial/boundary conditions (I/BCs). To overcome this issue, we propose an improved PINN model by utilizing multiple deep neural networks (DNNs) to construct the solution. Through a DNN pre-trained to represent the initial/boundary values, the approximated solution would obey the given I/BCs forcibly. With several numerical examples, we show that the improved PINN is characterized with much better accuracy on the I/BCs. Though the PINN shows great promise in data-driven modeling and solving inverse problems, some inherent limitations of PINN still exist, such as (i) the solution might lacks fine-scale details due to the global approximation of FCNN; (ii) high computational expense caused by the FCNN it roots on; (iii) incapability to incorporate existing PDE terms (e.g., $\Delta u$) into the network architecture. To overcome these drawbacks, this dissertation also proposes a discrete model - Physics-encoded Recurrent Convolutional Neural Network (PeRCNN) which recurrently updates the solution (or state variable) for time marching. Specifically, it utilizes convolutional neural network (CNN) to capture the spatial patterns of the solution while the recurrent network mimics the forward Euler scheme (or Runge-Kutta scheme) in numerical methods. PeRCNN is a mesh-based and discrete model due to the discretization in time and spatial dimension. The local connectivity of CNN makes PeRCNN more computationally efficient. In addition, the coercive encoding mechanism of physics in PeRCNN, fundamentally different from the PINN relying on soft penalty, ensures the network to rigorously obey given physics. The proposed PeRCNN is successfully applied to solving general PDEs, the data-driven modeling of dynamical systems and the data-driven discovery of governing PDEs from scarce and noisy measurements. Comparisons with the state-of-the-art DL models demonstrate that the proposed PeRCNN possesses excellent computational efficiency, accuracy and generalizability. --Author's abstract

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

Document classification is a prevalent task in Natural Language Processing (NLP), which has an extensive range of applications in the biomedical domains such as biomedical literature indexing, automatic diagnosis codes assignment, tweets classification for public health topics, and patient safety reports classification. Nevertheless, manual classification of biomedical articles published every year into specific predefined categories becomes a cumbersome task. Hence, building an automatic document classification for biomedical databases emerges as a significant task among the scientific community. In recent years, Deep Learning (DL) models like Deep Neural Networks (DNN), Convolution Neural Networks (CNN), Recurrent Neural Networks (RNN), and Ensemble Deep Learning models are widely used in the area of text document classification for better classification performance compared to Machine Learning (ML) algorithms. The major advantage of using DL models in document classification is that it provides rich semantic and grammatical information for document representation through pre-trained word embedding. Hence, this paper investigates the deployment of the various state-of-the-art DL based classification models in automatic classification of benchmark biomedical datasets. Finally, the performance of all the aforementioned constitutional classifiers is compared and evaluated through the well-defined performance evaluation metrics such as accuracy, precision, recall, and f1-measure.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

Machine Learning (ML) technologies have been widely used for blind signal classification problem (Modulation Classification) to automatically identify the modulation schemes of Radio Frequency (RF) signal from complex- valued IQ (In-phase Quadrature) samples. Traditional ML methods usually have two stages where the first stage is to manually extract the features of the IQ symbols by subject matter experts and the second stage is to feed the features to an ML algorithm (e.g., a support vector machine) to develop the classifier. The state- of-art technology is to apply Deep Learning (DL) technologies such as Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) directly to the complex-value IQ symbols to train a multi-class classifier. In this effort we are focused on the modulation and coding rate classification problems for multi-carrier Orthogonal Frequency-Division Multiplexing (OFDM) signals. The generated OFDM data set consists of over 480k instances across four different modulations and three different coding rates which include BPSK + 1=2, BPSK+3=4, QPSK+1=2, QPSK+3=4, QAM16+1=2, QAM16+3=4, QAM64+2=3 and QAM64+3=4. Deep Learning (DL) models can successfully catch the features of OFDM modulations and identify the modulation scheme from the baseband IQ symbols. Four DL models including Residual Neural Network 34(Resnet34), Resnet18, Squeezenet and Long Short-Term Memory (LSTM) trained over the data set can achieve over 94% overall accuracy for signals across 􀀀10dB to 10dB with step size of 4dB. Among them, Squeezenet and LSTM models have much smaller model sizes which can be easily loaded in resource-limited edge computing platforms. To the best of our knowledge, coding rate classification of OFDM signals has not been studied in previous works. Therefore, three different DL models including a single-stage coding rate classifier, a combination of modulation and coding rate classifier, and a two-stage coding rate classifier are developed to identify the coding rate. However, coding rates can only be identified for low-order modulation schemes BPSK and QPSK but not for high-order modulation schemes QAM16 and QAM64. Further investigation is needed to understand the coding rate classification for high-order QAM signals.

Prediction concept needs human-like thinking and ideas. ANN helps make a program for describing the solutions to the prediction problems. To gain accurate prediction, there is a need for deep thinking in the knowledge system discovered. Deep learning helps form such next-level type of ANN thinking. Deep learning models do not require humans for programming the problems. It can determine the prediction by itself by learning the dataset. This paper describes the prediction by considering sequential model for continuous accessing of data with the help of multi-layers. Deep learning is very helpful for predicting larger dataset. This paper proposes an algorithmic model known as a Comprehensive Deep Recurrent Artificial Neural Networks (CDRANN) by hybridizing LSTM (long short-term memory) with RNN (recurrent neural network) backtrack solver for better future stock prediction. This paper also explains the need for hybridizing evolutionary model CDRANN to create a novel change in future prediction.KeywordsCDRANN (Comprehensive Deep Recurrent Artificial Neural Networks)LSTM (long short-term memory)RNN (recurrent neural network)Stock marketPrediction

This article investigates the identification of hate speech on social media using machine learning and deep learning techniques. The research uses metrics such as F-measure, AUC-ROC, precision, accuracy, and recall assessing the effectiveness of various tactics. The findings indicate that deep learning models, particularly the bidirectional long short-term memory (BiLSTM) architecture, consistently outperform other methods in categorization tasks. The research emphasizes the importance of sophisticated neural network designs in identifying the intricacies of hostile and offensive content online. The study offers insights for promoting early identification and prevention of cyberbullying, improving secure and inclusive online environments. Future research may explore real-time detection systems, hybrid approaches, or the integration of complementary components to enhance and improve innovative technology in tackling this significant social issue. A sample tweet was annotated by specialists who categorize tweets as hate speech, offensive language, or neutral. The researchers applied shallow learning methodologies and integrated word embeddings like Word2Vec and GloVe to enhance the efficacy of deep learning models. The results indicate that BiLSTM surpasses shallow learning methods in detecting hate speech on Twitter, highlighting the efficacy of deep learning approaches in recognizing and tracking hate speech on social media platforms. When comparing different deep learning and machine learning models on different datasets, the results reveal that deep learning techniques are usually more effective. A reasonably high level of accuracy is achieved by KNN and SVM among classical algorithms, whereas Naïve Bayes performs the poorest. While deep learning approaches provide better results, tree-based models such as Random Forest and Decision Trees offer more consistent accuracy. Models based on neural networks, such as LSTM, CNN, and BI-LSTM, perform well, with LSTM-based methods excelling in particular. The most successful strategy for classification problems is the model presented, which obtains the greatest accuracy, precision, recall, F1-score of 95%. The research aids in the development of advanced tools and methodologies to mitigate hate speech on social media and foster positive online interactions. Future research may investigate alternative deep learning architectures, such as transformers, to enhance hate speech detection efficacy. The advancement of interpretable AI methodologies for identifying hate speech and delivering transparent forecasts might enhance user confidence and facilitate better content moderation decisions.

The exponential growth in the number of complex datasets every year requires\nmore enhancement in machine learning methods to provide robust and accurate\ndata classification. Lately, deep learning approaches have achieved surpassing\nresults in comparison to previous machine learning algorithms. However, finding\nthe suitable structure for these models has been a challenge for researchers.\nThis paper introduces Random Multimodel Deep Learning (RMDL): a new ensemble,\ndeep learning approach for classification. RMDL solves the problem of finding\nthe best deep learning structure and architecture while simultaneously\nimproving robustness and accuracy through ensembles of deep learning\narchitectures. In short, RMDL trains multiple randomly generated models of Deep\nNeural Network (DNN), Convolutional Neural Network (CNN) and Recurrent Neural\nNetwork (RNN) in parallel and combines their results to produce better result\nof any of those models individually. In this paper, we describe RMDL model and\ncompare the results for image and text classification as well as face\nrecognition. We used MNIST and CIFAR-10 datasets as ground truth datasets for\nimage classification and WOS, Reuters, IMDB, and 20newsgroup datasets for text\nclassification. Lastly, we used ORL dataset to compare the model performance on\nface recognition task.\n

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.