Efficient learning algorithms for neural networks (ELEANNE)

In the paper, the research present situation and development in the wavelet neural network model and a novel learning algorithm for wavelet neural network based on extended Kalman filter are discussed. Based on combining the exceptional property of localization of the wavelet transform and characteristics of self-learning of neural networks, the non-line time series model and network architecture model which combines affine transform with revolving transform is discussed .A novel learning algorithm for wavelet neural network based on extended Kalman filter is proposed to predict the deformation of structure. In comparison with the WNN algorithm, the EKF learning algorithm has improved convergence and can provide much more accuracy learning results.

Fractional calculus has been adopted in the modelling of many scientific processes and systems. Due to the inherent feature of long term memory of fractional derivatives, it has been used in the learning process of neural networks. A fractional order derivative based back propagation learning algorithm in neural networks is proposed in this paper. Specifically, Riemann-Liouville (R-L), Caputo (C) and Caputo Fabrizio (CF) fractional Derivative based on the back propagation algorithms in a three layer feed-forward neural network employed. To get a faster learning rate without oscillation, momentum factor is incorporated. The effect of fractional order and momentum factor is investigated and compared. The performance of these fractional derivatives based algorithms with integer derivatives based algorithm in terms of mean square error (MSE), particularly the salary based on years of experience is predicted. Results demonstrate that fractional derivative based learning algorithms outperform the integer derivatives.

The Sixth International Symposium on Neural Networks (ISNN 2009)

In this paper, a hybrid supervised learning algorithm for neural network was proposed. The problem of local minimum learning usually occurred in the real application of neural network is tried to be solved or reduced. In order to improve the efficiency and stability of conventional error back-propagation learning algorithm, a hybrid learning method combining the linear multi-regression and backpropagation techniques was developed. To demonstrate the superiority of the method we developed, one example was simulated. The conventional BP learning method was also performed as the comparison with the new method proposed. From the results shown, the conventional BP method easily makes neural model plunge into the local minima. On the contrary, the new method we proposed not only has a fast learning, but also has a better learning efficiency.

Extreme learning machine (ELM) is an easy-to use and effective learning algorithm of single-hidden layer feed-forward neural networks (SLFNs). The classical learning algorithm in neural network, e. g. backpropagation, requires setting several user-defined parameters and may get into local minimum. However, ELM only requires setting the number of hidden neurons and the activation function. It does not require adjusting the input weights and hidden layer biases during the implementation of the algorithm, and it produces only one optimal solution. Therefore, ELM has the advantages of fast learning speed and good generalization performance. In this paper, ELM is introduced in predicting reservoir permeability. By comparing to SVM, we analyze its feasibility and advantages in reservoir permeability prediction. The experimental results show that ELM has similar accuracy compared to SVR, but it has obvious advantages in parameter selection and learning speed.

The paper proposes a fast online learning method for neural network structures by using genetic algorithm (GA) and dynamic back propagation algorithm (BP) jointly. GA is used in the coarse tuning process which adjusts interconnection weights of the neural network. The dynamic back propagation algorithm is subsequently applied to achieve fine adjusting of the network weights. The fitness function, based on the squared error between the teaching signal and the network output value, is redefined at every time step and the proposed GA based algorithm solves a nonstationary function optimization task. At every time step the solution with the best fitness function is used for current representation of the neural network weights and biases. It is shown through the simulations and real time temperature control of drying oven that this learning algorithm has faster convergence ability and better performance on reducing mapping error in the online learning neural network structures. This leads to an improvement of the transient response of neuro adaptive systems. The proposed method has the potential to be applied to many practical areas such as system modeling and control, signal processing and pattern recognition.

Learning algorithm of neural network is always an important research contents in neural network theory research and application field, learning algorithm about the feed-forward neural network has no satisfactory solution in particular for its defects in calculation speed. The paper presents a new Fourier basis functions neural network algorithm and applied it to classify network trade customer. First, 21 customer classification indicators are designed, based on characteristics and behaviors analysis of network trade customer, including customer characteristics type variables and customer behaviors type variables,; Second, Fourier basis functions is used to improve the calculation flow and algorithm structure of original BP neural network algorithm to speed up its convergence and then a new Fourier basis neural network model is constructed. Finally the experimental results show that the problem of convergence speed can been solved, and the accuracy of the customer classification are ensured when the new algorithm is used in network trade customer classification practically. DOI: http://dx.doi.org/10.11591/telkomnika.v11i11.2978 Full Text: PDF

Nonlinear system control using self-evolving neural fuzzy inference networks with reinforcement evolutionary learning

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

A way in which a neural network can implement a genetic algorithm as its learning algorithm is shown. This model is called GLANN (genetic learning algorithm for neural networks). The components of GLANN can be shown to be biologically plausible. The algorithm itself can be classified as a reinforcement learning algorithm. The neural network has a fixed architecture and processes binary strings using genetic operators. Learning is stored in the form of newly created patterns, which can then be stored in some kind of associative memory. The benefits of GLANN reside in the proven optimizing capabilities of genetic algorithms, and in its parallel implementation. The shallow two-level architecture translates into system scalability, an issue that has not been successfully resolved in the case of other neural network algorithms. >

In this work, we propose a quantum algorithm to evaluate neural networks architectures named Quantum Neural Network Architecture Evaluation (QNNAE). The proposed algorithm is based on a quantum associative memory and the learning algorithm for artificial neural networks. Unlike conventional algorithms for evaluating neural network architectures, QNNAE does not depend on initialization of weights. The proposed algorithm has a binary output and results in 0 with probability proportional to the performance of the network. And its computational cost is equal to the computational cost to train a neural network.

Based on various approaches, several different learing algorithms have been given in the literature for neural networks. Almost all algorithms have constant learning rates or constant accelerative parameters, though they have been shown to be effective for some practical applications. The learning procedure of neural networks can be regarded as a problem of estimating (or identifying) constant parameters (i.e. connection weights of network) with a nonlinear or linear observation equation. Making use of the Kalman filtering, we derive a new back-propagation algorithm whose learning rate is computed by a time-varying Riccati difference equation. Perceptron-like and correlational learning algorithms are also obtained as special cases. Furthermore, a self-organising algorithm of feature maps is constructed within a similar framework.

Convergence theorems for the kohonen feature mapping algorithms with VLRPs

We estimate the spectrum envelope of speech by using neural networks. The neural networks system proposed in our study consists of two parts. The first part is the network for detecting the spectrum peaks, and the other is for interpolating between the extracted peaks. The structure of the latter is derived from the Neville interpolation. The former is based on the peak picking method. We present an operator for detecting the peaks using neural networks. Unfortunately, the result is that the neural networks do not only detect the general peaks but also the local peaks and the extremely closed peaks. We train the neural networks to detect only the general peaks. The learning algorithm uses a steepest descent method such as the backpropagation (BP) algorithm. After the learning, we show the detected peaks and the spectrum envelope when we use the learned pattern as the input data. In addition, we perform an experiment to examine whether the proposed operator corresponds to the XOR Boolean function or not.

Recent study, the extended Kalman filter (EKF) is applied to the learning algorithm for a feedforward neural network and it is shown that the EKF based learning algorithm has good convergence performances. However, the feedforward neural network could not perform a dynamical signal processing such as time series pattern recognition. On the other hand, the recurrent neural network (RNN) could have dynamical characteristics because the RNN has feedback connections with time delay in the network. The EKF based learning algorithms for the RNN are also reported and some training properties became clear. In this paper, RNN training based on the EKF are applied to the time series signal processing. The connection weights of the RNN can be modified by using the filtering algorithm based on the EKF. Simulation results show that the RNN based on the EKF has good performances of the training for time series patterns generated from sine functions and for some pulse patterns.