Recurrent neural networks: A constructive algorithm, and its properties

A real-coded genetic algorithm for training recurrent neural networks

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

In this paper, strategy on training recurrent neural networks based on genetic algorithms is expounded, which includes several key issues in designing a genetic algorithm, choice of running parameters and diagram of training method. Then forecasting nonlinear system using recurrent neural network is present. Simulation shows the method is effective.

Recurrent neural networks (RNNs) have emerged as a promising tool in modeling nonlinear dynamical systems. The convergence is one of the most important issues of concern among the dynamical properties for the RNNs in practical applications. The reason is that the viability of many applications of RNNs depends on their convergence properties. We study in this paper the convergence properties of the weighted state space search algorithm (WSSSA) -- a derivative-freeand non-random learning algorithm which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Because there is no computation of partial derivatives involved, the WSSSA has a couple of salient features such as simple,fast and cost effective. In this study we provide a necessary andsufficient condition that required for the convergence of the WSSSA. Restrictions are offered that may help assure convergence of the ofthe WSSSA to the desired solution. The asymptotic rate ofconvergence is also analyzed. Our study gives insights into theproblem and provides useful information for the actual design of theRNNs. A numerical example is given to support the theoreticalanalysis and to demonstrate that it is applicable to manyapplications.

New pruning algorithm for Recurrent Neural Networks

Positioning in real time with high precision is realized through implementing a Real-time Differential GPS (RTDGPS). Amccuracy of the system depends on sequential and uninterruptible transmission of Pseudo-Range Correction (PRC) codes from Reference Station (RS) to the user station. This transmission takes place in RTCM protocol. In order to compensate the signal absence and transmission delay of differential corrections, future values of the PRC should somehow be estimated. To this end a combination of Recurrent Neural Network (RNN) and Genetic Algorithm (GA) has been used. Addition of this algorithm not only improved the efficiency and consistency of the RTDGPS service, but also increased the accuracy of the positioning system. In this research RTDGPS system has been implemented using two inexpensive receivers in reference and user stations. Simulation and experimental results indicate that RTDGPS accuracy improved by predicting the future values of PRC. Time step of the prediction algorithm is considered to be 5 s. RMS value of the positioning error for a fixed point after addition of the RNN-GA algorithm to RTDGPS system has decreased.

IMDb is a well-known platform that provides user reviews and ratings of various movies. The number of reviews found on IMDb is quite large, reaching thousands of reviews. Although a movie can have a high overall rating, it is still possible to receive negative reviews from some viewers. Therefore, the purpose of this sentiment classification system is to provide a benchmark for the level of sentiment contained in the movie, and hope that filmmakers can use this information as a reference in the development of their next movie. In this research, reviews from IMDb users are classified into two types, namely positive reviews and negative reviews. The program was created using the Python language with the LSTM (Long Short-Term Memory) classification model of the RNN (Recurrent Neural Network) algorithm. The purpose of using this algorithm is to measure the level of prediction accuracy in the classification process. The results of three test ratios, namely 60:40, 70:30, and 80:20, show that in the scenario of 80% data training and 20% data testing has better performance with the results accuracy of 96%, precision of 97%, recall of 98%, f1-score of 97%.

On the improvement of the real time recurrent learning algorithm for recurrent neural networks

This paper discusses the use of a recent boosting algorithm for recurrent neural networks as a tool to model nonlinear dynamical systems. It combines a large number of RNNs, each of which is generated by training on a different set of examples. This algorithm is based on the boosting algorithm where difficult examples are concentrated on during the learning process. However, unlike the original algorithm, all examples available are taken into account. The ability of the method to internally encode useful information on the underlying process is illustrated by several experiments on well known chaotic processes. Our model is able to find an appropriate internal representation of the underlying process from the observation of a subset of the states variables. We obtain improved prediction performances.

The Industrial Internet of Things (IIoT) integrated with cloud computing resources offers effective advancements in the field of industrial automation with open connectivity and emergent computing. With such advancements, the supplementary services evolves in order to augment the automation process in manufacturing and production industries. However, the integration and transformation of cloud and IIoT environment poses serious limitation on reliability and computational performance. It is very necessary to develop a framework that handles and provides a bridge between the rapid input acquisitions from an IIoT device and rapid data processing or storage by the cloud. In this paper, model based engineering (MBE) model is used to control the workflow in cloud environment, which is integrated with a deep learning algorithm named recurrent neural network (RNN) algorithm. The RNN is utilized to control the future workload prediction and supply necessary resources to balance the workflow in cloud computing environment. The entire predictions are carried out based on the input data acquisition rate from the IIoT devices. The storage or processing of the input data is carried out by the Virtual Machines presents in cloud environment. The resources allocation and scheduling of workload is carried out in the control plane using RNN that optimizes the workflow. The RNN is resumed to solve iteratively the task of optimization. The experiments are conducted to test the efficacy of the model and the results are verified in terms of total energy consumption, task response time, and VM Utilization. The experiments are conducted with same/different types of IIoT devices. The simulation results reveal the improved efficacy of the MBE–RNN model than existing deep learning and MBE schedulers. The results show that the MBE–RNN acquires improved processing ability with reduced task response time and energy utilization.

The segmentation of the left ventricle (LV) wall in four-chamber view cardiac sequential image is significant for cardiac disease diagnosis and cardiac mechanisms study; however, there is no successful reported work on sequential four-chambered view LV wall segmentation due to the complex four-chamber structure and diversity of wall motion. In this article, we propose a dense recurrent neural network (RNN) algorithm to achieve accurately LV wall segmentation in a four-chamber view MRI time sequence. In the cardiac sequential LV wall process, not only the sequential accuracy but also the accuracy of each image matters. Thus, we propose a dense RNN to provide compensation for the first long short-term memory (LSTM) cells. Two RNNs are combined in this work, the first one aims at providing information for the first image, and the second RNN generates segmentation result. In this way, the proposed dense RNN improves the accuracy of the first frame image. What is more is that, it improves the effectiveness of information flow between LSTM cells. Obtaining more competent information from the former cell, frame-wise segmentation accuracy is greatly improved. Based on the segmentation result, an algorithm is proposed to estimate cardiac state. This is the first time that deals with both cardiac time-sequential LV segmentation problems and, robustly, estimates cardiac state. Rather than segmenting each frame separately, utilizing cardiac sequence information is more stable. The proposed method ensures an Intersection over Union (IoU) of 92.13%, which outperforms other classical deep learning algorithms.

This paper presents an enhanced machine learning based state of charge (SOC) estimation method for a lithium-ion battery using a deep recurrent neural network (DRNN) algorithm. DRNN is suitable for SOC evaluation due to strong computation intelligence and self-learning capabilities. Nevertheless, the performance of DRNN is constrained due to the training accuracy and duration which entirely depends on the appropriate selection of hyper-parameters including hidden layer and hidden neurons. Therefore, firefly algorithm (FA) is employed to find the optimal number for hyper-parameters of DRNN networks. The optimized DRNN based FA algorithm for SOC estimation does not require extensive knowledge about battery chemistry, electrochemical battery model and added filter, rather only needs battery test bench to measure current and voltage. The developed model is tested using two different types of lithium-ion batteries namely lithium nickel manganese cobalt oxide (LiNiMnCoO 2 ) and lithium nickel cobalt aluminum oxide (LiNiCoAlO 2 ). The proposed model is validated by two experimental tests; one with static discharge test and other with pulse discharge test at room temperature. The experimental results indicate the superiority of the DRNN based FA method in comparison with the back-propagation neural network (BPNN) and radial basis function neural network (RBFNN).

Euro 6 is the latest vehicle emission standards for pollutants such as CO, NO2 and PM, that all new vehicles must comply, and it was introduced in September 2015 in South Korea. This study examined the effect of Euro 6 by comparing the measured pollutant concentrations after 2016 (Euro 6–era) to the estimated concentrations without Euro 6. The concentration without Euro 6 was estimated by first modeling the air quality using various environmental factors related to diesel vehicles, meteorological conditions, temporal information such as date and precursors in 2002–2015 (pre–Euro 6–era), and then applying the model to predict the concentration after 2016. In this study, we used both recurrent neural network (RNN) and random forest (RF) algorithms to model the air quality and showed that RNN can achieve higher R2 (0.634 ~ 0.759 depending on pollutants) than RF, making it more suitable for air quality modeling. According to our results, the measured concentrations during 2016–2019 were lower than the concentrations predicted using RNN by − 1.2%, − 3.4%, and − 4.8% for CO, NO2 and PM10. Such reduction can be attributed to the result of Euro 6.

Summary A meticulous interpretation of steady-state or unsteady-state relative permeability (Kr) experimental data is required to determine a complete set of Kr curves. In this work, different machine learning (ML) models were developed to assist in a faster estimation of these curves from steady-state drainage coreflooding experimental runs. These ML algorithms include gradient boosting (GB), random forest (RF), extreme gradient boosting (XGB), and deep neural network (DNN) with a main focus on and comparison of the two latter algorithms (XGB and DNN). Based on existing mathematical models, a leading-edge framework was developed where a large database of Kr and capillary pressure (Pc) curves were generated. This database was used to perform thousands of coreflood simulation runs representing oil-water drainage steady-state experiments. The results obtained from these simulation runs, mainly pressure drop along with other conventional core analysis data, were used to estimate analytical Kr curves based on Darcy’s law. These analytically estimated Kr curves along with the previously generated Pc curves were fed as features into the ML model. The entire data set was split into 80% for training and 20% for testing. The k-fold cross-validation technique was applied to increase the model’s accuracy by splitting 80% of the training data into 10 folds. In this manner, for each of the 10 experiments, nine folds were used for training and the remaining fold was used for model validation. Once the model was trained and validated, it was subjected to blind testing on the remaining 20% of the data set. The ML model learns to capture fluid flow behavior inside the core from the training data set. In terms of applicability of these ML models, two sets of experimental data were needed as input; the first was the analytically estimated Kr curves from the steady-state drainage coreflooding experiments, while the other was the Pc curves estimated from centrifuge or mercury injection capillary pressure (MICP) measurements. The trained/tested model was then able to estimate Kr curves based on the experimental results fed as input. Furthermore, to test the performance of the ML model when only one set of experimental data is available to an end user, a recurrent neural network (RNN) algorithm was trained/tested to predict Kr curves in the absence of Pc curves as an input. The performance of the three developed models (XGB, DNN, and RNN) was assessed using the values of the coefficient of determination (R2) along with the loss calculated during training/validation of the model. The respective crossplots along with comparisons of ground truth vs. artificial intelligence (AI)-predicted curves indicated that the model is capable of making accurate predictions with an error percentage between 0.2% and 0.6% on history-matching experimental data for all three tested ML techniques. This implies that the AI-based model exhibits better efficiency and reliability in determining Kr curves when compared to conventional methods. The developed ML models by no means replace the need to conduct drainage coreflooding or centrifuge experiments but act as an alternative to existing commercial platforms that are used to interpret experimental data to predict Kr curves. The two main advantages of the developed ML models are their capability of predicting Kr curves within a matter of a few minutes as well as with limited intervention from the end user. The results also include a comparison between classical ML approaches, shallow neural networks, and DNNs in terms of accuracy in predicting the final Kr curves. The research presented here is an extension of the state-of-the-art framework proposed by Mathew et al. (2021). However, the two main aspects of the current study are the application of deep learning for the prediction of Kr curves and the application of feature engineering. The latter not only reduces the training/testing time for the ML models but also enables the end user to obtain the final predictions with the least set of experimental data. The various models discussed in this research work currently focus on the prediction of Kr curves for drainage steady-state experiments; however, the work can be extended to capture the imbibition cycle as well.

The abnormal growth of human healthy cells is called cancer. One of the major types of cancer is sarcoma, mostly found in human bones and soft tissue cells. It commonly occurs in children. According to a survey of the United States of America, there are more than 17,000 sarcoma patients registered each year which is 15% of all cancer cases. Recognition of cancer at its early stage saves many lives. The proposed study developed a framework for the early detection of human sarcoma cancer using deep learning Recurrent Neural Network (RNN) algorithms. The DNA of a human cell is made up of 25,000 to 30,000 genes. Each gene is represented by sequences of nucleotides. The nucleotides in a sequence of a driver gene can change which is termed as mutations. Some mutations can cause cancer. There are seven types of a gene whose mutation causes sarcoma cancer. The study uses the dataset which has been taken from more than 134 samples and includes 141 mutations in 8 driver genes. On these gene sequences RNN algorithms Long and Short-Term Memory (LSTM), Gated Recurrent Units and Bi-directional LSTM (Bi-LSTM) are used for training. Rigorous testing techniques such as Self-consistency testing, independent set testing, 10-fold cross-validation test are applied for the validation of results. These validation techniques yield several metrics such as Area Under the Curve (AUC), sensitivity, specificity, Mathew's correlation coefficient, loss, and accuracy. The proposed algorithm exhibits an accuracy of 99.6% with an AUC value of 1.00.