Stochastic Resonance Enables BPP/log* Complexity and Universal Approximation in Analog Recurrent Neural Networks

The Sixth International Symposium on Neural Networks (ISNN 2009)

A G/sub m/C-style state constrained neuron (SCN) model for the design of processors in analog recurrent neural networks such as Hopfield neural networks, cellular nonlinear networks for combinatorial optimization is described. The unconstrained neurons which have the free state variable, could be stable at any arbitrary point in the solution space or trapped by un-intentional effects. These may introduce errors. For the unconstrained network, the solution could be different from the expected one due to the discrepancy in the energy function of the network and the objective function to be optimized. In addition, if the state variable is limited by some neighboring saturated transistors, un-desirable results may be obtained. The G/sub m/C-style SCN model can ensure the convergence of the network and avoid discrepancy between the energy function of the network and the objective function. The state resistor is also eliminated in the G/sub m/C model so that high cell-density can be achieved. Simulation results show that the proposed model is effective in significantly reducing optimization error.

The universal approximation theorem asserts that a single hidden layer neural network approximates continuous functions with any desired precision on compact sets. As an existential result, the universal approximation theorem supports the use of neural networks for various applications, including regression and classification tasks. The universal approximation theorem is not limited to real-valued neural networks but also holds for complex, quaternion, tessarines, and Clifford-valued neural networks. This paper extends the universal approximation theorem for a broad class of hypercomplex-valued neural networks. Precisely, we first introduce the concept of non-degenerate hypercomplex algebra. Complex numbers, quaternions, and tessarines are examples of non-degenerate hypercomplex algebras. Then, we state the universal approximation theorem for hypercomplex-valued neural networks defined on a non-degenerate algebra.KeywordsHypercomplex algebrasNeural networksUniversal approximation theorem

In this paper, (supra-threshold) stochastic resonance phenomenon of noise-enhanced information transmission is studied in detail through the numerical calculation and the computer simulation in a non-linear multilevel threshold neural networks system, which is affected by both additive noise and multiplicative noise, then the mutual information is used to characterize the phenomenon. The mutual information as a function of additive noise intensity or multiplicative noise intensity brings on convex changes under a suitable system threshold and a fixed multiplicative noise intensity or additive noise intensity, which shows that the (supra-threshold) stochastic resonance phenomenon occurs. The increases in the number of the system threshold units can enhance the effectiveness of information transmission; the increase of the system threshold can increase the signal components that are under the threshold, and thus the supra-threshold stochastic resonance takes place more easily. In addition, by changing the additive noise intensity the supra-threshold stochastic resonance occurs more easily than by changing the multiplicative noise intensity. The above results show that both the existence of the supra-threshold stochastic resonance and the effectiveness of noise-improved the signal transmission are closely related to multiplicative or additive noise intensity, the number of threshold units, and the system threshold level.

Given N data pairs {X i , y i }, i=1,2,...,N, where each X i is an n-dimensional vector of independent variables \((X_i = )\) and y i is a dependent variable, the function approximation problem (FAP) is finding a function that best explains the N pairs of X i and y i . From the universal approximation theorem and inherent approximation capabilities proved by various researches, artificial neural networks (ANNs) are considered as powerful function approximators. There are two main issues on the feedforward neural networks’ performance. One is to determine its structure. The other issue is to specify the weights of a network that minimizes its error. Genetic algorithm (GA) is a global search technique and is useful for complex optimization problems. So, it has been considered to have potential to reinforce the performance of neural networks. Many researchers tried to optimize the weights of networks using genetic algorithms alone or combined with the backpropagation algorithm. Others also tried to find a good topology that is even more difficult and called a “black art.”

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

What does it mean when we say that computers can ‘write’ and how are recent developments in neural networks and machine learning changing this capacity? This article examines the long-standing literary fear of authorship being replaced by machines while also interrogating the labour and credit implications that sit behind widely used structures of authorship in a technological age. The argument makes reference to one work of computer-generated writing – Johannes Helden & Hakan Jonson’s Evolution [2014] – and to one software paradigm (a character-based recurrent neural networks for language acquisition trained on the corpus of the journal Textual Practice). I here argue that unless we conceive more broadly of the criteria for ‘authorship’ as a labour function, and unless we take seriously the need to see textual production as social production, hybridized (but predominantly) machine identities will come to dominate a literary landscape.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

Large scale artificial neural networks (ANNs) have been widely used in data processing applications. The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Such a unique architecture enables the recurrent neural network to remember the past processed information and makes it an expressive model for nonlinear sequence processing tasks. However, the large computation complexity makes it difficult to effectively train a recurrent neural network and therefore significantly limits the research on the recurrent neural network in the last 20 years. In recent years, the use of graphics processing units (GPUs) becomes a significant advance to speed up the training process of large scale neural networks by taking advantage of the massive parallelism capabilities of GPUs. In this paper, we propose an efficient GPU implementation of the large scale recurrent neural network and demonstrate the power of scaling up the recurrent neural network with GPUs. We first explore the potential parallelism of the recurrent neural network and propose a fine-grained two-stage pipeline implementation. Experiment results show that the proposed GPU implementation can achieve 2 ~ 11 x speed-up compared with the basic CPU implementation with the Intel Math Kernel Library. We then use the proposed GPU implementation to scale up the recurrent neural network and improve its performance. The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset.

In the field of big data analytics, deep learning has garnered a lot of interest due to the fact that it is capable of doing feature extraction and classification. The categorization of big data has traditionally been carried out by academics via the use of Machine Learning methods; nonetheless, the process of feature extraction has traditionally been carried out manually. Deep learning is a method developed by researchers that allows for the extraction of features via the use of various algorithms. This research made use of an algorithm known as "Convolutional Neural Network," or CNN for short, in order to get a deeper comprehension of the dependability of big data analytics. Primary research has been conducted in order to get an understanding of how hidden nodes and layers influence the accuracy of this neural network. The "Recurrent Neural Network" (also known as RNN) and the "Artificial Neural Network" (also known as ANN) are the two other neural network algorithms that have been selected for comparison with CNN in order to evaluate whether or not CNN is more effective than the other algorithms. In the first research, the reliability of CNN as well as other methods was analyzed in relation to the number of nodes, the number of hidden layers, the amount of training time, and the amount of validating time. Control variables were employed in both correlation and regression investigations. These variables included Hidden Nodes, Training time, Valid time, and Secret Layers. CNN, ANN, and RNN were significant variables. According to the findings, CNN is more dependable than other neural networks for analyzing huge amounts of data, scoring 92%, whereas other neural networks scored less than 90%. The inclusion of hidden nodes results in a significant increase in CNN's reliability.

Artificial neural networks allow the identification of black-box models. This paper proposes a method aimed at replicating the static and dynamic behavior of a DC-DC power converter based on a recurrent nonlinear autoregressive exogenous neural network. The method proposed in this work applies an algorithm that trains a neural network based on the inputs and outputs (currents and voltages) of a Buck converter. The approach is validated by means of simulated data of a realistic nonsynchronous Buck converter model programmed in Simulink and by means of experimental results. The predictions made by the neural network are compared to the actual outputs of the system, to determine the accuracy of the method, thus validating the proposed approach. Both simulation and experimental results show the feasibility and accuracy of the proposed black-box approach.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

Research in the area of artificial neural networks has advanced at a rapid pace in recent times. The artificial neural network possesses a good ability to learn adaptively. The decision process in an artificial neural network is based on certain nonlinear operations. Such nonlinearities are useful: i) in improving the convergence speed (of the algorithm); ii) to provide more general nonlinear mapping between input-output signals; and iii) to reduce the effect of outliers in the measurements. One of the most successful artificial neural networks is the so-called feed forward neural network. In this chapter first the description of the feed forward neural network and its training algorithms is given. Next, parameter estimation using this approach is discussed. The presentation of training algorithms is such that it facilitates MATLAB implementation. Subsequently, recurrent neural networks are described. Several schemes based on recurrent neural networks are presented for parameter estimation of dynamical systems. Subsequently, the genetic algorithm is described and its application to parameter estimation considered.

Application of the diagonal recurrent wavelet neural network to solar irradiation forecast assisted with fuzzy technique

Neural Networks, with their remarkable capacity to get significance from convoluted information can be utilized to remove patterns that are too composite to be in any way seen by humans. A prepared neural network can be thought of as a specialist in the classification of data which is given to analyze. There are different kinds of Neural Networks like Artificial Neural Network (ANN), Feedforward Neural Network, Recurrent Neural Network(RNN), Recursive Recurrent Neural Network (RRNN), Convolutional Neural Network(CNN), Modular Neural Network (MNN), Restricted Boltzmann Machine (RBM) etc. In this paper, we have discussed the performance of ANN, CNN, RNN, and RBM where CNN has outplayed the remaining with accuracy of 97.81%.