DiffRNN: Differential Verification of Recurrent Neural Networks

Prediction concept needs human-like thinking and ideas. ANN helps make a program for describing the solutions to the prediction problems. To gain accurate prediction, there is a need for deep thinking in the knowledge system discovered. Deep learning helps form such next-level type of ANN thinking. Deep learning models do not require humans for programming the problems. It can determine the prediction by itself by learning the dataset. This paper describes the prediction by considering sequential model for continuous accessing of data with the help of multi-layers. Deep learning is very helpful for predicting larger dataset. This paper proposes an algorithmic model known as a Comprehensive Deep Recurrent Artificial Neural Networks (CDRANN) by hybridizing LSTM (long short-term memory) with RNN (recurrent neural network) backtrack solver for better future stock prediction. This paper also explains the need for hybridizing evolutionary model CDRANN to create a novel change in future prediction.KeywordsCDRANN (Comprehensive Deep Recurrent Artificial Neural Networks)LSTM (long short-term memory)RNN (recurrent neural network)Stock marketPrediction

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

Recurrent neural networks (RNN) are at the core of modern automatic speech recognition (ASR) systems. In particular, long short-term memory (LSTM) recurrent neural networks have achieved state-of-the-art results in many speech recognition tasks, due to their efficient representation of long and short term dependencies in sequences of inter-dependent features. Nonetheless, internal dependencies within the element composing multidimensional features are weakly considered by traditional real-valued representations. We propose a novel quaternion long short-term memory (QL-STM) recurrent neural network that takes into account both the external relations between the features composing a sequence, and these internal latent structural dependencies with the quaternion algebra. QLSTMs are compared to LSTMs during a memory copy-task and a realistic application of speech recognition on the Wall Street Journal (WSJ) dataset. QLSTM reaches better performances during the two experiments with up to 2.8 times less learning parameters, leading to a more expressive representation of the information.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

Artificial Intelligence (AI) has recently emerged as a powerful tool with versatile applications spanning various domains. AI replicates human intelligence processes through machinery and computer systems, finding utility in expert systems, image and speech recognition, machine vision, and natural language processing (NLP). One notable area with limited exploration pertains to using deep learning models, specifically Recurrent Neural Networks (RNNs), for predicting water quality in wastewater treatment plants (WWTPs). RNNs are purpose-built for handling sequential data, featuring a feedback mechanism. However, standard RNNs may exhibit limitations in accommodating both short-term and long-term dependencies when addressing intricate time series problems. The solution to this challenge lies in adopting Long Short-Term Memory (LSTM) cells, known for their inherent memory management through a ‘forget gate’ mechanism. In general, LSTM architecture demonstrates superior performance. WWTP data represent a historical series influenced by fluctuating environmental conditions. This study employs simple RNNs and LSTM architecture to construct prediction models for effluent parameters, systematically assessing their performance through various training data scenarios and model architectures. The primary objective was to determine the most suitable WWTP dataset model. The study revealed that an epoch setting of 50 and a batch size of 100 yielded the lowest training time and root mean square error (RMSE) values for both RNN and LSTM models. Furthermore, when these models are applied to predict effluent parameters, they exhibit precise RMSE values for all parameters. The study results can be applied to detect potential upsets in WWTP operations.

One of the most promising tools in recent years for the analysis and prediction of time series data, which includes financial market data has been the use of neural networks. To ensure the accuracy of the outcomes of these systems, it is critical to overcome the vanishing gradient and exploding gradient problems that often occur when recurrent neural networks (RNN) process data. Long Short-Term Memory (LSTM) has been shown to provide good performance when dealing with time series datasets. This paper will explore the feasibility of using an RNN with LSTM as a predictive tool for use with futures trading data. Using a dataset comprised of all futures trading occurring on the Bourse de MontrA©al (TMX) during a 9-month period from January to September 2015, we assessed the predictive effectiveness of an RNN in predicting the price of front-end contracts for the futures symbol BAX. We found that while an RNN provided a degree of short-term predictive capability, this capability did not extend beyond a couple of days. Although it failed as a trading instrument to predict futures prices, the RNN could detect, identify, and reflect underlying trends in the data, indicating the tool may hold promise in the detection of trading patterns.Â

While the growing accessibility of technology improves usability, it also creates more opportunities for cybercriminals to exploit vulnerabilities, significantly accelerating the proliferation of cybersecurity attacks. Deep learning (DL) approaches present significant advancements over conventional machine learning (ML) techniques by automating feature selection and extraction while minimizing external dependencies. This study proposes a deep learning-based model to enhance cyberattack detection and ensure that data security goals are achieved. This study employs a quantitative research design, utilizing simulation and modeling as the primary analytical tools. The dataset used is the Canadian Institute for Cybersecurity Intrusion Detection System (CIC-IDS-2017) dataset. Three distinct DL algorithms are used to design the detection models, namely, Deep Neural Network (DNN), Long Short-Term Memory (LSTM), and Recurrent Neural Network (RNN). The performance comparison metrics are F1-score, accuracy, sensitivity, false positive rate (FPR), specificity, positive predictive value (PPV), false negative rate (FNR), and negative predictive value (NPV). Optimization concepts are integrated to enhance the detection efficiency in web-based systems, including loss functions, gradient-based optimization, and efficient model generalization techniques. The results of k-fold cross-validation show LSTM’s higher scores for F1-score (94.6%), recall (94.7%), accuracy (94.8%), and precision (94.6%). LSTM outperformed RNN and DNN, achieving the highest accuracy, precision, specificity, and sensitivity at 94.7%, 94.3%, 98.9%, and 94.7% respectively, validating LSTM's superior generalizability for cyberattack detection tasks.

Internet of Things (IoT) technology has been incorporated into the majority of people’s everyday lives and places of employment due to the quick development in information technology. Modern agricultural techniques increasingly use the well-known and superior approach of managing a farm known as “smart farming”. Utilizing a variety of information and agricultural technologies, crops are observed for their general health and productivity. This requires monitoring the condition of field crops and looking at many other indicators. The goal of smart agriculture is to reduce the amount of money spent on agricultural inputs while keeping the quality of the final product constant. The Internet of Things (IoT) has made smart agriculture possible through data collection and storage techniques. For example, modern irrigation systems use effective sensor networks to collect field data for the best plant irrigation. Smart agriculture will become more susceptible to cyber-attacks as its reliance on the IoT ecosystem grows, because IoT networks have a large number of nodes but limited resources, which makes security a difficult issue. Hence, it is crucial to have an intrusion detection system (IDS) that can address such challenges. In this manuscript, an IoT-based privacy-preserving anomaly detection model for smart agriculture has been proposed. The motivation behind this work is twofold. Firstly, ensuring data privacy in IoT-based agriculture is of the utmost importance due to the large volumes of sensitive information collected by IoT devices, including on environmental conditions, crop health, and resource utilization data. Secondly, the timely detection of anomalies in smart agriculture systems is critical to enable proactive interventions, such as preventing crop damage, optimizing resource allocation, and ensuring sustainable farming practices. In this paper, we propose a privacy-encoding-based enhanced deep learning framework for the difficulty of data encryption and intrusion detection. In terms of data encoding, a novel method of a sparse capsule-auto encoder (SCAE) is proposed along with feature selection, feature mapping, and feature normalization. An SCAE is used to convert information into a new encrypted format in order to prevent deduction attacks. An attention-based gated recurrent unit neural network model is proposed to detect the intrusion. An AGRU is an advanced version of a GRU which is enhanced by an attention mechanism. In the results section, the proposed model is compared with existing deep learning models using two public datasets. Parameters such as recall, precision, accuracy, and F1-score are considered. The proposed model has accuracy, recall, precision, and F1-score of 99.9%, 99.7%, 99.9%, and 99.8%, respectively. The proposed method is compared using a variety of machine learning techniques such as the deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM).

In this study, we investigate the effectiveness of various deep learning algorithms for customer segmentation in the banking sector, aiming to enhance targeted service delivery and customer experience. We employ a comprehensive pipeline encompassing data collection, preprocessing, feature selection, feature extraction, model development, and rigorous evaluation. Our dataset, derived from real-world banking customer profiles, was processed using normalization, encoding, and dimensionality reduction techniques. We implemented and compared eight models: Artificial Neural Network (ANN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), CNN-LSTM Hybrid, Autoencoder-Based Network, and Deep Neural Network (DNN). Among them, the Autoencoder-Based model achieved the highest accuracy of 91.56%, outperforming others in terms of segmentation clarity and computational efficiency. These findings suggest that deep learning methods, particularly Autoencoder-Based architectures, offer robust solutions for real-time banking customer segmentation, enabling institutions to tailor products and services more effectively.

Stock price returns forecasting is challenging task for day traders to yield more returns. In the past, most of the literature was focused on machine learning algorithm to predict the stock returns. In this work, the recurrent neural network (RNN) with long short term memory (LSTM) is studied to forecast future stock returns. It has the ability to keep the memory of historical stock returns in order to forecast future stock return output. RNN with LSTM is used to store recent stock information than old related stock information. We have considered a recurrent dropout in RNN layers to avoid overfitting in the model. To accomplish the task we have calculated stock return based on stock closing prices. These stock returns are given as input to the recurrent neural network. The objective function of the prediction model is to minimize the error in the model. To conduct the experiment, data is collected from the National Stock Exchange, India (NSE). The proposed RNN with LSTM model outperforms compared to an feed forward artificial neural network.

Robust recurrent neural networks for time series forecasting

Trajectory-level fog detection based on in-vehicle video camera with TensorFlow deep learning utilizing SHRP2 naturalistic driving data

Recurring wildfires pose a critical global issue as they undermine social and economic stability and jeopardize human lives. To effectively manage disasters and bolster community resilience, the development of wildfire susceptibility maps (WFSMs) has emerged as a crucial undertaking in recent years. In this research endeavor, two deep learning algorithms were leveraged to generate WFSMs using two distinct remote sensing datasets. Specifically, the Moderate-Resolution Imaging Spectroradiometer (MODIS) and Landsat-8 images were utilized to monitor wildfires that transpired during the year 2021. To develop an effective WFSM, two datasets were created by incorporating 599 wildfire locations with Landsat-8 images and 232 sites with MODIS images, as well as twelve factors influencing wildfires. Deep learning algorithms, namely the long short-term memory (LSTM) and recurrent neural network (RNN), were utilized to model wildfire susceptibility using the two datasets. Subsequently, four WFSMs were generated using the LSTM (MODIS), LSTM (Landsat-8), RNN (MODIS), and RNN (Landsat-8) algorithms. The evaluation of the WFSMs was performed using the area under the receiver operating characteristic (ROC) curve (AUC) index. The results revealed that the RNN (MODIS) (AUC = 0.971), RNN (Landsat-8) (AUC = 0.966), LSTM (MODIS) (AUC = 0.964), and LSTM (Landsat-8) (AUC = 0.941) algorithms demonstrated the highest modeling accuracy, respectively. Moreover, the Gini index was employed to assess the impact of the twelve factors on wildfires in the study area. The results of the random forest (RF) algorithm indicated that temperature, wind speed, slope, and topographic wetness index (TWI) parameters had a significant effect on wildfires in the study region. These findings are instrumental in facilitating efficient wildfire management and enhancing community resilience against the detrimental effects of wildfires.

Over the past few decades, research has focused heavily on automatic speech recognition (ASR). Although ASR for a few languages is close to reality; ASR for the low resource languages like Malayalam is still in its infancy. Here, in this work, the authors discuss the experiment conducted on accented data of Malayalam speech using two approaches. One approach uses the spectral features for modeling using deep convolutional neural network and the other uses the influential features of speech signals for modeling using LSTM-RNN approach. The proposed methodology comprises three distinct stages; dataset preparation, feature extraction, classification, and hence the construction of deep learning models that recognize the accent-based spoken sentences in the Malayalam language. Mel-frequency cepstral coefficient (MFCC) algorithm, short-term Fourier transform (STFT), and mel spectrogram methodologies are used for feature engineering and hence the features that represent the speech signals are used for constructing the accented ASR system for the Malayalam language using long short-term memory (LSTM) a recurrent neural network (RNN). The spectrogram dataset has been constructed for the speech dataset and used for constructing the ASR model with deep convolutional neural network (DCNN). The result of the experiment shows that LSTM-based RNN outperforms DCNN for the proposed dataset that has been constructed for the experiment in the natural recording environment.

Recurrent neural networks (RNNs) can be used to handle sequential patterns and have been used for speech recognition. To overcome the shortcomings of RNN, recurrent sub neural networks (RSNNs) are used, where an RSNN is built independently for each class. The training algorithm of the RSNN is based on the backpropagation algorithm. Speaker dependent connected Chinese digit-speech recognition experiments were carried out. Some factors influencing the performance of RSNNs have been studied. The experiments show that RSNN is easier to train and gives higher performance than RNN.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>