Reinforcement learning method for continuous state space based on dynamic neural network

Recurrent neural networks unlike feed-forward networks are able to process inputs with time context. The key role in this process is played by the dynamics of the network, which transforms input data to the recurrent layer states. Several authors have described and analyzed dynamics of small sized recurrent neural networks with two or three hidden units. In our work we introduce techniques that allow to visualize and analyze the dynamics of large recurrent neural networks with dozens units, reveal both stable and unstable points (attractors and saddle points), which are important to understand the principles of successful task processing. As a practical example of this approach, dynamics of the simple recurrent network trained by two different training algorithms on context-free language anbnwas studied.

This paper deals with the adaptive nonlinear identification and trajectory tracking via dynamic multilayer neural network (NN) with different timescales. Two NN identifiers are proposed for nonlinear systems identification via dynamic NNs with different timescales including both fast and slow phenomenon. The first NN identifier uses the output signals from the actual system for the system identification. In the second NN identifier, all the output signals from nonlinear system are replaced with the state variables of the NNs. The online identification algorithms for both NN identifier parameters are proposed using Lyapunov function and singularly perturbed techniques. With the identified NN models, two indirect adaptive NN controllers for the nonlinear systems containing slow and fast dynamic processes are developed. For both developed adaptive NN controllers, the trajectory errors are analyzed and the stability of the systems is proved. Simulation results show that the controller based on the second identifier has better performance than that of the first identifier.

The article is devoted to the problem of optimization of search request ranking algorithms in the digital information retrieval system. The algorithm of functioning of the neural network ranking unit based on Hopfield neural network is built. The ability to generate a ranked list of pages found as a result of the request in the digital information retrieval system can be provided by solving two problems of integer optimization: the problem of assignment of combinatorial sets of criteria for assessing the relevance of web page search and the problem of sorting of numbers—relevance values. The architecture of the neural network model based on the dynamic Hopfield neural network with binary output function designed for combinatorial optimization of the final list of documents found in the digital information retrieval system was synthesized. Promising variants of neural network models with binary output function of neurons for synthesis of the optimal evaluation plan with a combinatorial set of criteria by solving the problem of assignment were built. It has been proven that the built models differ in the rules for determining the coefficients of synaptic connections and external shifts; each of the created rules can be used independently or in different combinations with one another. In the course of analytical research, it was found that the optimization formulation of the problem of sorting of relevance values of search pages is identical to the problem of assignment of combinatorial groups of evaluation criteria provided that the elements of the performance matrix of the latter are defined as linear combinations of relevance values.

This is a continuation of a recent study (Doruk RO, Zhang K. Fitting of dynamic recurrent neural network models to sensory stimulus-response data. J Biol Phys 2018; 44: 449-469), where a continuous time dynamical recurrent neural network is fitted to neural spiking data. In this research, we address the issues arising from the inclusion of sigmoidal gain function parameters to the estimation algorithm. The neural spiking data will be obtained from the same model as that of Doruk and Zhang, but we propose a different model for identification. This will also be a continuous time recurrent neural network, but with generic sigmoidal gains. The simulation framework and estimation algorithms are kept similar to that of Doruk and Zhang so that we can have a solid base to compare the results. We evaluate the estimation performance in two different ways. First, we compare the firing rate responses of the original and the estimated model. We find that responses of both models to the same stimuli are similar. Secondly, we evaluate variations of the standard deviations of the estimates against a number of samples and stimulus parameters. They show a similar pattern to that of Doruk and Zhang. We thus conclude that our model serves as a reasonable alternative provided that firing rate is the response of interest (to any stimulus).

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

Fault detection and diagnosis for non-linear processes empowered by dynamic neural networks

Several mathematical approaches to studying analytically the dynamics of neural networks rely on mean-field approximations, which are rigorously applicable only to networks of infinite size. However, all existing real biological networks have finite size, and many of them, such as microscopic circuits in invertebrates, are composed only of a few tens of neurons. Thus, it is important to be able to extend to small-size networks our ability to study analytically neural dynamics. Analytical solutions of the dynamics of small-size neural networks have remained elusive for many decades, because the powerful methods of statistical analysis, such as the central limit theorem and the law of large numbers, do not apply to small networks. In this article, we critically review recent progress on the study of the dynamics of small networks composed of binary neurons. In particular, we review the mathematical techniques we developed for studying the bifurcations of the network dynamics, the dualism between neural activity and membrane potentials, cross-neuron correlations, and pattern storage in stochastic networks. Then, we compare our results with existing mathematical techniques for studying networks composed of a finite number of neurons. Finally, we highlight key challenges that remain open, future directions for further progress, and possible implications of our results for neuroscience.

An adaptive identification method of simple dynamic recurr ent neural network (SRNN) for nonlinear dynamic systems is presented in this paper. This method based on the theory that by using the inner-states feed-back of dynamic network to describe the nonlinear kinetic characteris tics of system can reflect the dynamic ch aracteristics more directly, deduces the recursive prediction error (RPE) learning algorithm of SRNNe and improves the algorithm by studying topological structure on recursion layer without the weight values. The simulation results indicate that this kind of neural network can be used in real-time control, due to its less weight values, simpler learning algorithm, higher identification speed, and higher precision of model. It solves the problems of intricate in training algorithm and slow rate in convergence caused by the complicate topologi cal structure in usual dynami c recurrent neural network. Keywords: non-linear system, system identification, dynamic recurrent neural network, RPE algorithm

An increasing number of applications of dynamic neural networks has been developed for digital signal processing (DSP), dynamic neural networks are feedforward neural networks with commonly used scalar synapses replaced by linear filters. This provides feedforward neural networks with the capability of performing dynamic mappings, which depend on past input values, dynamic neural networks are suitable for time series prediction, nonlinear system identification, and signal processing applications. Their most popular types are Finite Impulse Response (FIR) neural networks, which are obtained by replacing synapses with finite impulse response filters. Due to their guaranteed stability characteristic and easy to minimize error surface they have been used with great success in many applications such as signal enhancement, noise cancellation, classification of input patterns, system identification, prediction, and control.. Most of the works on system identification using neural networks are based on multilayer feedforward neural networks with backpropagation learning or more efficient variations of this algorithm, an elegant method for training layered networks. This paper is based on work in a Dynamic System Modeling (DYSMO) and as an application for speed control of DC motor drive.

Dynamic graph neural network with adaptive edge attributes for air quality prediction: A case study in China

In this paper, a three-dimensional spatial-temporal decomposition modelling method is proposed to build the alkali-surfactant-polymer (ASP) flooding model, in which a new dynamic recurrent wavelet neural network (DRWNN) is presented to identify the temporal coefficients. At first, the detailed mathematical model of ASP flooding is described which is a complex distributed parameter system. Then a three-dimensional spatial-temporal modelling method is inferred based on Karhunen-Loeve (K-L) decomposition to decompose the water saturation of reservoir into a series of spatial basis functions and corresponding temporal coefficients. Furthermore, the recurrent wavelet neural network is used to acquire the identification model, in which the injection concentrations of ASP flooding and temporal coefficients are taken as the input and output information. In order to improve the capability of dynamic modelling, DRWNN is proposed through adding feedback layers and setting the different weights with time to achieve dynamic memory of the past information. Considering the gradient descent method for the neural networks training easily leads to local minimum and slow convergence speed, the spectral conjugate gradient method is introduced to optimize the weights of DRWNN. At last, DRWNN is used to build the relation between the moisture content of production wells and the water saturation of the corresponding grids. Thus, the final approximate model of ASP flooding is finished. The accuracy is proved by model with four injection wells and nine production wells through data from the mechanism model.

Online news recommendation aims to continuously select a pool of candidate articles that meet the temporal dynamics of user preferences. Most of the existing methods assume that all user-item interaction history are equally importance for recommendation, which is not alway applied in real-word scenario since the user-item interactions are sometime full of stochasticity and contingency. In addition, previous work on session-based algorithms only considers user sequence behaviors within current session without incorporating users’ historical interests or pointing out users’ main purposes within such session. In this paper, we propose a novel neural network framework, dynamic attention-integrated neural network, to tackle the problems. Specifically, we propose a dynamic neural network to model users’ dynamic interests over time in a unified framework for personalized news recommendations. News article semantic embedding, user interests modelling, session-based public behavior mining and an attention scheme that used to learn the attention score of user and item interaction within sessions are four key factors for online sequences mining and recommendation strategy. Experimental results on three real-world datasets show significant improvements over several baselines and state-of-the-art methods on session-based neural networks.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

Model predictive control of an industrial packed bed reactor using neural networks

Intelligent Multivariable Modeling of Blast Furnace Molten Iron Quality Based on Dynamic AGA-ANN and PCA