Constructive training of recurrent neural networks using hybrid optimization

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

The main task of using neural networks is the prompt and accurate solution of various creative tasks, including the analysis and synthesis of news flows, while maintaining the continuity of learning. The result of such processing can be digests, filtered news streams, as well as event forecasts that allow for proactivity in management decisions. Known methods of news processing by neural networks and technical solutions that implement them do not fully provide a solution to the problems that arise in this area. It is necessary to expand their functionality, and improve the space-time signal binding in recurrent neural networks. When processing news flows, simultaneously with continuous training of recurrent neural networks, selection, recognition, restoration, prediction and synthesis of news should be carried out. To reduce the severity of the problem, a promising method of multifunctional processing of news flows is proposed using recurrent neural networks with a logical organization of layers and continuous learning. The method is based on the development of associative processing of textual information in streaming recurrent neural networks with controlled elements. The key features of this method are the multifunctional processing of information flows with changing laws of news appearance. The method provides for operational selection, recognition, restoration, forecasting and synthesis of news based on deep associative continuous processing of links between text elements. The neural network system that implements the proposed method differs from the known solutions by new elements, connections between them, as well as by the functions performed. The results of the experiments confirmed the extended functionality of the method. New features of processing news texts by streaming RNNs are revealed. The proposed solutions can be used to create a new generation of intelligent systems not only for word processing, but also for other types of information.

We investigate how unsupervised training of recurrent neural networks (RNNs) and their deep hierarchies can benefit a supervised task like temporal pattern detection. The RNNs are fully and fast trained by unsupervised algorithms and only supervised feed-forward readouts are used. The unsupervised RNNs are shown to perform better in a rigorous comparison against state-of-art random reservoir networks. Unsupervised greedy bottom-up trained hierarchies of such RNNs are shown being capable of big performance improvements over single layer setups.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

Unintentional falls can cause severe injuries and even death, especially if no immediate assistance is given. A fall detection system aims to detect a fall as soon as it occurs, therefore issuing an automatic assistance request. Wearable embedded sensors are emerging as the most viable solution for continuous monitoring since they are more effective, less intrusive and less expensive than other systems. Tailoring a deep learning method to the requirements of microcontrollers entails matching very stringent constraints in terms of both memory and computational power. In addition, datasets acquired with wearable devices are relatively scarce and not necessarily devised for supervised learning. In this work, we discuss the design of a software architecture based on recurrent neural networks which can be effective for fall detection while running entirely onboard a wearable device. The well-known and publicly-available SisFall dataset was adopted and extended with fine-grained temporal annotations to cope with the supervised training of recurrent neural networks. We then show that an appropriate process of architectural minimization together with accurate hyperparameters selection leads to a workable model which compares favorably with other detection techniques. The embedding of the resulting architecture has been validated using a state-of-art hardware device.

In this paper we demonstrate a modified topology for long short-term memory recurrent neural networks that controls the shape of the squashing functions in gating units. We further propose an efficient training framework based on a mini-batch training on sequence level combined with a sequence chunking approach. The framework is evaluated on publicly available data sets containing English and French handwriting by utilizing a GPU based implementation. Speedups of more than 3x are achieved in training recurrent neural network models which outperform state of the art recognition results.

Training of neural network can be accelerated by limited numerical precision together with specialized low-precision hardware. This paper studies how low precision can impact on entire training of RNNs. We emulate low precision training for recently proposed gated recurrent unit (GRU) and use dynamic fixed point as a target numeric format. We first show that batch normalization on input sequences can help speed up training with low precision as well as high precision. We also show that the overflow rate should be carefully controlled for dynamic fixed point. We study low precision training with various rounding options including bit truncation, round to nearest, and stochastic rounding. Stochastic rounding shows superior results than the other options. The effect of fully low precision training is also analyzed by comparing partial low precision training. We show that the piecewise linear activation function with stochastic rounding can achieve comparable training results with floating point precision. Low precision multiplier and accumulator (MAC) with linear-feedback shift register (LFSR) is implemented with 28nm Synopsys PDK for energy and performance analysis. Implementation results show low precision hardware is 4.7x faster, and energy per task is up to 4.55x lower than that of floating point hardware.

Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.

Reservoir Computing (RC) is a paradigm for efficient training of Recurrent Neural Networks (RNNs). The Echo State Network (ESN), a type of RC paradigm, has been widely used for time series forecasting. Whereas, few works exist on classification with ESN. In this paper, we shed light on the use of ESN for pattern recognition problem, i.e. emotion recognition from Electroencephalogram (EEG). We show that the reservoir with its recurrence is able to perform the feature extraction step directly from the EEG raw. Such kind of recurrence rich of nonlinearities allows the projection of the input data into a high dimensional state space. It is well known that the ESN fails due to the poor choices of its initialization. Nevertheless, we show that pretraining the ESN with the Intrinsic Plasticity (IP) rule remedies the shortcoming of randomly initialization. To validate our approach, we tested our system on the benchmark DEAP containing EEG signals of 32 subjects and the results were promising.

Modeling of gene regulatory networks with hybrid differential evolution and particle swarm optimization

A combination of Cuckoo Search algorithm and Recurrent Neural Network is proposed in this paper to track the maximum power from solar photovoltaic system under complex partial shading conditions with minimum tracking speed. Cuckoo Search is a meta-heuristic search algorithm which imitates the behavior of the cuckoo bird. The Recurrent Neural Network is a highly successful technique in nonlinear systems. It can interpolate and extrapolate the random data with high accuracy if the network is properly trained. There are two stages in the proposed hybrid technique. In stage 1 (Off-line stage), Cuckoo Search algorithm identifies the maximum power under different solar radiation patterns and converts them into control pulses. The control pulses are given as target and the solar radiation of individual photovoltaic module as input to train the Recurrent Neural Network, also the weights are optimized. In stage 2 (On-line stage), the trained Recurrent Neural Network is tested for real time conditions. The proposed hybrid algorithm is implemented in MATLAB/Simulink platform and the tracking performance is compared with the state-of-art methods.

Task-trained artificial recurrent neural networks (RNNs) provide a computational modeling framework of increasing interest and application in computational, systems, and cognitive neuroscience. RNNs can be trained, using deep-learning methods, to perform cognitive tasks used in animal and human experiments and can be studied to investigate potential neural representations and circuit mechanisms underlying cognitive computations and behavior. Widespread application of these approaches within neuroscience has been limited by technical barriers in use of deep-learning software packages to train network models. Here, we introduce PsychRNN, an accessible, flexible, and extensible Python package for training RNNs on cognitive tasks. Our package is designed for accessibility, for researchers to define tasks and train RNN models using only Python and NumPy, without requiring knowledge of deep-learning software. The training backend is based on TensorFlow and is readily extensible for researchers with TensorFlow knowledge to develop projects with additional customization. PsychRNN implements a number of specialized features to support applications in systems and cognitive neuroscience. Users can impose neurobiologically relevant constraints on synaptic connectivity patterns. Furthermore, specification of cognitive tasks has a modular structure, which facilitates parametric variation of task demands to examine their impact on model solutions. PsychRNN also enables task shaping during training, or curriculum learning, in which tasks are adjusted in closed-loop based on performance. Shaping is ubiquitous in training of animals in cognitive tasks, and PsychRNN allows investigation of how shaping trajectories impact learning and model solutions. Overall, the PsychRNN framework facilitates application of trained RNNs in neuroscience research.

In order to improve the accuracy and stability of English translation, this paper proposes an English translation model based on recurrent neural network. Based on the end-to-end encoder-decoder architecture, a recursive neural network (RNN) English machine translation model is designed to promote machine autonomous learning features, transform the distributed corpus data into word vectors, and directly map the source language and target language through the recurrent neural network. Selecting semantic errors to construct the objective function during training can well balance the influence of each part of the semantics and fully consider the alignment information, providing a strong guidance for the training of deep recurrent neural networks. The experimental results show that the English translation model based on recurrent neural network has high effectiveness and stability. Compared with the baseline system, it has improved about 1.51–1.86 BLEU scores. Conclusion. The model improves the performance and quality of English machine translation model, and the translation effect is better.

Singing voice conversion is a task to convert a song sang by a source singer to the voice of a target singer. In this paper, we propose using a parallel data free, many-to-one voice conversion technique on singing voices. A phonetic posterior feature is first generated by decoding singing voices through a robust Automatic Speech Recognition Engine (ASR). Then, a trained Recurrent Neural Network (RNN) with a Deep Bidirectional Long Short Term Memory (DBLSTM) structure is used to model the mapping from person-independent content to the acoustic features of the target person. F0 and aperiodic are obtained through the original singing voice, and used with acoustic features to reconstruct the target singing voice through a vocoder. In the obtained singing voice, the targeted and sourced singers sound similar. To our knowledge, this is the first study that uses non parallel data to train a singing voice conversion system. Subjective evaluations demonstrate that the proposed method effectively converts singing voices.

Existing AI systems already have the capabilities to transform the way diseases are diagnosed and treated. ParkinsonAI is no exception—we propose a solution to the cumbersome and inaccessible methods currently used to diagnose Parkinson’s Disease. These diagnostic methods either rely on specialized medical equipment or require trained professionals to operate, making prompt diagnosis at home impossible. Since Parkinson’s disease cannot be cured, treatments stem solely from its management. As such, the early diagnosis of Parkinson’s disease is crucial as it allows for the initiation of treatment plans at an earlier stage, which can significantly improve the overall quality of life for patients. This study introduces ParkinsonAI, an AI-powered system designed for the early identification of potential Parkinson’s Disease (PD) related motor dysfunctions. ParkinsonAI leverages Google’s Mediapipe framework to analyze hand motion through 21 key points, capturing intricate skeletal movements. This data from the hand is then fed into our trained Recurrent Neural Network (RNN) to report the severity of the condition. To train the AI model, 62 user datasets were collected, with 19 used for model validation. The final confusion matrix revealed an 84.2% accuracy, though it potentially may be influenced by limited data for specific severity levels. Future research will focus on refining and expanding ParkinsonAI’s capabilities to enhance its accuracy, sensitivity, and clinical utility.