Bidirectional Quaternion Long Short-term Memory Recurrent Neural Networks for Speech Recognition

This article explores the suitability of a long short-term memory recurrent neural network (LSTM-RNN) and artificial intelligence (AI) method for low-flow time series forecasting. The long short-term memory works on the sequential framework which considers all of the predecessor data. This forecasting method used daily discharged data collected from the Basantapur gauging station located on the Mahanadi River basin, India. Different metrics [root-mean-square error (RMSE), Nash–Sutcliffe efficiency (ENS), correlation coefficient (R) and mean absolute error] were selected to assess the performance of the model. Additionally, recurrent neural network (RNN) model is also used to compare the adaptability of LSTM-RNN over RNN and naive method. The results conclude that the LSTM-RNN model (R = 0.943, ENS = 0.878, RMSE = 0.487) outperformed RNN model (R = 0.935, ENS = 0.843, RMSE = 0.516) and naive method (R = 0.866, ENS = 0.704, RMSE = 0.793). The finding of this research concludes that LSTM-RNN can be used as new reliable AI technique for low-flow forecasting.

With the development of China's Belt and Road Initiative (BRI), the port plays a significant role and its operation management faces some pressure. In this regard, prediction of daily container volumes will provide the manager with data support for better plan of a storage yard. In this work, by deep learning the historical dataset, the long short-term memory (LSTM) recurrent neural network (RNN) is trained and used to predict daily volumes of containers which will enter the storage yard. The raw dataset of a certain port from 2013 to 2016 is chosen as the training set and the dataset of 2017 is used as the test set to evaluate the performance of the proposed prediction model. Then the LSTM model is established with Python and Tensorflow framework. The structure parameters are adjusted to find the optimal LSTM network, so as to improve the prediction accuracy. It appears that the LSTM model with two hidden layers and 30 hidden layer units has less prediction error between the real data and predicted data of 2017. The prediction error of daily container volumes between predicted value and real data of 2017 is about 12.39%, which is less than the people-predicted error. It is promising that the proposed LSTM RNN model can be applied to predict the daily volumes of containers and have higher prediction accuracy.

Remaining useful life prediction of PEMFC based on long short-term memory recurrent neural networks

Maxout neurons for deep convolutional and LSTM neural networks in speech recognition

Capturing long-range interactions between structural but not sequence neighbors of proteins is a long-standing challenging problem in bioinformatics. Recently, long short-term memory (LSTM) networks have significantly improved the accuracy of speech and image classification problems by remembering useful past information in long sequential events. Here, we have implemented deep bidirectional LSTM recurrent neural networks in the problem of protein intrinsic disorder prediction. The new method, named SPOT-Disorder, has steadily improved over a similar method using a traditional, window-based neural network (SPINE-D) in all datasets tested without separate training on short and long disordered regions. Independent tests on four other datasets including the datasets from critical assessment of structure prediction (CASP) techniques and >10 000 annotated proteins from MobiDB, confirmed SPOT-Disorder as one of the best methods in disorder prediction. Moreover, initial studies indicate that the method is more accurate in predicting functional sites in disordered regions. These results highlight the usefulness combining LSTM with deep bidirectional recurrent neural networks in capturing non-local, long-range interactions for bioinformatics applications. SPOT-disorder is available as a web server and as a standalone program at: http://sparks-lab.org/server/SPOT-disorder/index.php . j.hanson@griffith.edu.au or yuedong.yang@griffith.edu.au or yaoqi.zhou@griffith.edu.au. Supplementary data is available at Bioinformatics online.

Long Short Term Memory (LSTM) Recurrent Neural Networks has been shown to be capable of learning long time dependencies, and has been successfully applied to many studies, such as machine translation, speech recognition and air pollution concentration prediction. The present research has shown that the presence of missing data could dramatically degrade the results of data mining and categorical predictions with the aid of the machine learning technique including LSTM networks. Therefore, this paper focuses on imputation of missing data in the time series of air pollutants using LSTM networks to improve the PM2.5 concentration prediction accuracy. Experimental result shows that the proposed LSTM-based imputation method presents better PM2.5 concentration prediction accuracy than mean-imputation method and moving average imputation method.

Recurrent neural networks (RNNs) are naturally suitable for speech recognition because of their ability of utilizing dynamically changing temporal information. Deep RNNs have been argued to be able to model temporal relationships at different time granularities, but suffer vanishing gradient problems. In this paper, we extend stacked long short-term memory (LSTM) RNNs by using grid LSTM blocks that formulate computation along not only the temporal dimension, but also the depth dimension, in order to alleviate this issue. Moreover, we prioritize the depth dimension over the temporal one to provide the depth dimension more updated information, since the output from it will be used for classification. We call this model the prioritized Grid LSTM (pGLSTM). Extensive experiments on four large datasets (AMI, HKUST, GALE, and MGB) indicate that the pGLSTM outperforms alternative deep LSTM models, beating stacked LSTMs with 4% to 7% relative improvement, and achieve new benchmarks among uni-directional models on all datasets.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

This work aims to develop an automatic tonal and non-tonal language classification system of Indian languages using cascade Convolutional Neural Network (CNN)-Long Short-Term Memory (LSTM) Recurrent neural networks (RNNs). Motivated by their success in modelling sequences, this study proposes LSTM-RNNs cascaded with CNN in context of tonal and non-tonal language classification. Here RNNs show its effectiveness to exploit temporal dependencies in acoustic data. This paper also proposes the use of pitch chroma spectrogram coefficients to address this classification task. The proposed feature is then combined with log-mel spectrogram coefficients to enhance the system performance. The system has been tested for NIT Silchar language database (NITS-LD) which is developed for 9 Indian languages using All India radio broadcast news. And it reports accuracies of 82.30% for 10s and 81.16% for 3s data. Performance of the proposed system is also analyzed on Oregon Graduate Institute Multi-Language Telephone-based Speech (OGI-MLTS) database. It shows accuracies of 77.2% and 74.95% for 10s and 3s data respectively.

Photovoltaic power generation is one of the most efficient renewable energy generation techniques. To ensure photovoltaic (PV) safe operation and cost-effective integration into smart grids. As a result, a realistic PV power forecast method can aid in mitigating PV power generation’s drawbacks, which is critical for power plant maintenance and repair. In this research, we compared two Deep Learning models which are both highly advantageous, namely a long short-term memory (LSTM) and a recurrent neural network (RNN). A case study employing an actual dataset obtained from 1B DKASC, Alice Springs, Australia, is used to demonstrate the performance of LSTM and RNN. The forecasts generated by the LSTM are compared with the results of RNN using MAE, MSE, MBE, R\(^2\) and RMSE for all the forecast horizons. The results of the study demonstrate that the LSTM based forecast outperforms RNN and has the potential to improve the accuracy and stability of the prediction.KeywordsDeep LearningLong Short-Term Memory (LSTM)Recurrent neural network (RNN)

Ensemble of long short-term memory recurrent neural network for semi-active control of tuned liquid wall damper

The accurate estimation of energy expenditure from simple objective accelerometry measurements provides a valuable method for investigating the effect of physical activity (PA) interventions or population surveillance. Methods have been evaluated previously, but none utilize the temporal aspects of the accelerometry data. In this study, we investigated the energy expenditure prediction from acceleration measured at the subjects’ hip, wrist, thigh, and back using recurrent neural networks utilizing temporal elements of the data. The acceleration was measured in children (N = 33) performing a standardized activity protocol in their natural environment. The energy expenditure was modelled using Multiple Linear Regression (MLR), stacked long short-term memory (LSTM) networks, and combined convolutional neural networks (CNN) and LSTM. The correlation and mean absolute percentage error (MAPE) were 0.76 and 19.9% for the MLR, 0.882 and 0.879 and 14.22% for the LSTM, and, with the combined LSTM-CNN, the best performance of 0.883 and 13.9% was achieved. The prediction error for vigorous intensities was significantly different (p < 0.01) from those of the other intensity domains: sedentary, light, and moderate. Utilizing the temporal elements of movement significantly improves energy expenditure prediction accuracy compared to other conventional approaches, but the prediction error for vigorous intensities requires further investigation.

This paper proposes a multi-layered anomaly detection scheme to train feature extraction and to test anomaly prediction by using Convolutional Neural Networks (CNNs) layer, Bidirectional and Unidirectional Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNNs), which is one of a novel deep architecture named stacked convolutional bidirectional LSTM network (SCB-LSTM). In the proposed model, the stacked CNNs perform feature extraction of vibration sensor signal patterns, and the result is used to feature learning with the stacked bidirectional LSTMs (SB-LSTMs). After this procedure, the stacked unidirectional LSTMs (SU-LSTMs) enhance the feature learning, and a regression layer finally predicts anomaly detections. The experimental results of bearing data not only show the accuracy of the proposed model in anomaly detection for rotating machinery diagnostics, but also suggest the better performance than other state-of-the-art algorithms such as a plain uni-LSTM or Bi-LSTM.

This paper develops a general framework for learning interpretable data representation via Long Short-Term Memory (LSTM) recurrent neural networks over hierarchal graph structures. Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization. We thus call this model the structure-evolving LSTM. In particular, starting with an initial element-level graph representation where each node is a small data element, the structure-evolving LSTM gradually evolves the multi-level graph representations by stochastically merging the graph nodes with high compatibilities along the stacked LSTM layers. In each LSTM layer, we estimate the compatibility of two connected nodes from their corresponding LSTM gate outputs, which is used to generate a merging probability. The candidate graph structures are accordingly generated where the nodes are grouped into cliques with their merging probabilities. We then produce the new graph structure with a Metropolis-Hasting algorithm, which alleviates the risk of getting stuck in local optimums by stochastic sampling with an acceptance probability. Once a graph structure is accepted, a higher-level graph is then constructed by taking the partitioned cliques as its nodes. During the evolving process, representation becomes more abstracted in higher-levels where redundant information is filtered out, allowing more efficient propagation of long-range data dependencies. We evaluate the effectiveness of structure-evolving LSTM in the application of semantic object parsing and demonstrate its advantage over state-of-the-art LSTM models on standard benchmarks.

Short-term traffic flow prediction plays an important role in intelligent transportation system. Numerous researchers have paid much attention to it in the past decades. However, the performance of traditional traffic flow prediction methods is not satisfactory, for those methods cannot describe the complicated nonlinearity and uncertainty of the traffic flow precisely. Neural networks were used to deal with the issues, but most of them failed to capture the deep features of traffic flow and be sensitive enough to the time-aware traffic flow data. In this paper, we propose a deep bi-directional long short-term memory (DBL) model by introducing long short-term memory (LSTM) recurrent neural network, residual connections, deeply hierarchical networks and bi-directional traffic flow. The proposed model is able to capture the deep features of traffic flow and take full advantage of time-aware traffic flow data. Additionally, we introduce the DBL model, regression layer and dropout training method into a traffic flow prediction architecture. We evaluate the prediction architecture on the dataset from Caltrans Performance Measurement System (PeMS). The experiment results demonstrate that the proposed model for short-term traffic flow prediction obtains high accuracy and generalizes well compared with other models.