Complex-Valued Feedforward Neural Networks Learning Without Backpropagation

A fast and robust algorithm for training feedforward neural networks (FNNs) by using a variable forgetting factor and U-D factorization-based recursive prediction error (RPE) method is proposed. In comparison with the backpropagation (BP) and RPE based learning algorithms, the proposed algorithm, called UD-RPE, can provide much more accurate learning results in fewer iterations with fewer hidden nodes and improve convergence rate and numerical stability (robustness). In addition, it is less sensitive to start-up parameters, such as initial weights and initial covariance matrix, and the randomness in the observed data. It also has good generalization ability and needs less learning time. Simulation results of nonlinear dynamic system modeling and identification show that the algorithm proposed here is an effective and efficient learning algorithm for FNNs.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

Learning algorithms in neural networks: recent results

In this paper, the terminal attractor based back propagation learning algorithms for feedforward neural networks are examined. Through a rigorous mathematical analysis, a condition to guarantee the convergence of the algorithms is given. A simulation study is presented to demonstrate the effectiveness of the analysis.

The convergence of a class of stochastic pseudo-gradient algorithms driven by correlated data sequences is considered in this paper. The obtained results are applied to a learning algorithm for feedforward neural networks and sufficient conditions for its convergence are determined.

Linear programming algorithm for neural networks

Based on the critical position of the endpoint quality prediction for basic oxygen furnaces (BOFs) in steelmaking, and the latest results in computational intelligence (CI), this paper deals with the development of a novel memetic algorithm (MA) for neural network (NN) learning. Included in this is the integration of extremal optimization (EO) and Levenberg-Marquardt (LM) gradient search, and its application in BOF endpoint quality prediction. The fundamental analysis reveals that the proposed EO-LM algorithm may provide superior performance in generalization, computation efficiency, and avoid local minima, compared to traditional NN learning methods. Experimental results with production-scale BOF data show that the proposed method can effectively improve the NN model for BOF endpoint quality prediction.

Complex-valued neural network is a type of neural networks, which is extended from real number domain to complex number domain. Fully complex extreme learning machine (CELM) is an efficient algorithm, which owes faster convergence than the common complex backpropagation (CBP) neural networks. However, it needs more hidden neurons to reach competitive performance. Recently, an efficient learning algorithm is proposed for the single-hidden layer feed-forward neural network which is called the upper-layer-solution-aware algorithm (USA). Motivated by USA, an efficient algorithm for complex-valued neural networks through training input weights (GGICNN) has been proposed to train the split complex-valued neural networks in this paper. Compared with CELM and CBP, an illustrated experiment has been done in detail which observes the better generalization ability and more compact architecture for the proposed algorithm.

A new fast learning algorithm for a multilayer feedforward neural network is presented. The new algorithm, based on an innovative variable step size and gradient averaging, has excellent properties to overcome major drawbacks of the backpropagation algorithm and shows high convergence speed. The improved performance of the new algorithm in comparison with other algorithms is verified via computer simulations.

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

Many real scenarios in machine learning are non-stationary. These challenges forces to develop new algorithms that are able to deal with changes in the underlying problem to be learnt. These changes can be gradual or abrupt. As the dynamics of the changes can be different, the existing machine learning algorithms exhibit difficulties to cope with them. In this work we propose a new method, that is based in the introduction of a forgetting function in an incremental online learning algorithm for two-layer feedforward neural networks. This forgetting function gives a monotonically crescent importance to new data. Due to this fact, the network forgets in presence of changes while maintaining a stable behavior when the context is stationary. The theoretical basis for the method is given and its performance is illustrated by evaluating its behavior. The results confirm that the proposed method is able to work in evolving environments.

In order to solve the problem that the traditional long-term high-speed traffic forecasting algorithm is affected by the approximation ability of the function and easy to fall into the local mass value, we wrote a multivariate-based highway traffic forecasting algorithm scaling and convolutional networks. Because the feedforward wavelet neural network algorithm predicts the short-term traffic flow in different areas, it is necessary to examine the ability to predict the difference between different models. From the standard feedforward wavelet neural network algorithm using global optimization capabilities, we improve the wolf pack algorithm, improve the search accuracy of the algorithm, get the best solution of the estimated value of the work according to the search results when completing the research objectives, and get the ability to predict the work of the model. Feedforward neural network algorithm: we develop and obtain the best short-term high-speed traffic forecast values. The results are as follows: after using the author’s algorithm, the processing time increases by 1.5 seconds, but the average percentage of errors decreases by more than 50%, in fact the error and the root mean square error decreased by about 30%, and the smoothing coefficient increased by about 1%. The prediction of the author’s algorithm for short-term high-speed traffic is better than the wavelet neural network prediction algorithm, and the prediction accuracy and stability of the author’s algorithm are higher.

Neural network model has undergone many modifications from the original model through the development of network architecture and optimization methods. By default, the gradient-based optimization method is a method used to find network weights. But some of the weaknesses and limitations of these methods inspired many researchers to try to use other methods. Non-gradient based heuristic method is a reasonable choice considering the learning algorithm in artificial neural networks is inspired by the characteristics of creatures so that optimization methods that also mimic life patterns in nature will be appropriate. One disadvantage of the heurisic method is the length of the iteration process. In this paper, a method which combines heuristic optimization methods and gradient-based methods is applied, namely particle swarm optimization (PSO) and Levenberg Marquardt. The weight obtained from the PSO method becomes the initial weight for the Levenberg Marquardt method. The proposed procedure is applied to the rainfall data in Cokrotulung Klaten. The results showed that this procedure succeeded in providing better predictions than the Levenberg Marquardt method.

Fractional calculus has been adopted in the modelling of many scientific processes and systems. Due to the inherent feature of long term memory of fractional derivatives, it has been used in the learning process of neural networks. A fractional order derivative based back propagation learning algorithm in neural networks is proposed in this paper. Specifically, Riemann-Liouville (R-L), Caputo (C) and Caputo Fabrizio (CF) fractional Derivative based on the back propagation algorithms in a three layer feed-forward neural network employed. To get a faster learning rate without oscillation, momentum factor is incorporated. The effect of fractional order and momentum factor is investigated and compared. The performance of these fractional derivatives based algorithms with integer derivatives based algorithm in terms of mean square error (MSE), particularly the salary based on years of experience is predicted. Results demonstrate that fractional derivative based learning algorithms outperform the integer derivatives.

Learning algorithm of neural network is always an important research contents in neural network theory research and application field, learning algorithm about the feed-forward neural network has no satisfactory solution in particular for its defects in calculation speed. The paper presents a new Fourier basis functions neural network algorithm and applied it to classify network trade customer. First, 21 customer classification indicators are designed, based on characteristics and behaviors analysis of network trade customer, including customer characteristics type variables and customer behaviors type variables,; Second, Fourier basis functions is used to improve the calculation flow and algorithm structure of original BP neural network algorithm to speed up its convergence and then a new Fourier basis neural network model is constructed. Finally the experimental results show that the problem of convergence speed can been solved, and the accuracy of the customer classification are ensured when the new algorithm is used in network trade customer classification practically. DOI: http://dx.doi.org/10.11591/telkomnika.v11i11.2978 Full Text: PDF