Novel fast training algorithm for multilayer feedforward neural network

A fast and robust algorithm for training feedforward neural networks (FNNs) by using a variable forgetting factor and U-D factorization-based recursive prediction error (RPE) method is proposed. In comparison with the backpropagation (BP) and RPE based learning algorithms, the proposed algorithm, called UD-RPE, can provide much more accurate learning results in fewer iterations with fewer hidden nodes and improve convergence rate and numerical stability (robustness). In addition, it is less sensitive to start-up parameters, such as initial weights and initial covariance matrix, and the randomness in the observed data. It also has good generalization ability and needs less learning time. Simulation results of nonlinear dynamic system modeling and identification show that the algorithm proposed here is an effective and efficient learning algorithm for FNNs.

Presents an algorithm of dynamic node creation and weights learning for a hybrid feedforward neural network (HFNN) which consists of a linear model and a multilayer neural network. The algorithm is only based on linear least squares, and no iterative learning process is needed. According to the demand of model precision, the algorithm determines the best weights of network and the minimal number of hidden nodes automatically. Compared with the well-known backpropagation network, simulation results show that the new learning algorithm for the HFNN is efficient in model precision, rate of convergence and generalization ability.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

In this paper a general class of fast learning algorithms for feedforward neural networks is introduced and described. The approach exploits the separability of each layer into linear and nonlinear blocks and consists of two steps. The first step is the descent of the error functional in the space of the outputs of the linear blocks (descent in the neuron space), which can be performed using any preferred optimization strategy. In the second step, each linear block is optimized separately by using a least squares (LS) criterion. To demonstrate the effectiveness of the new approach, a detailed treatment of a gradient descent in the neuron space is conducted. The main properties of this approach are the higher speed of convergence with respect to methods that employ an ordinary gradient descent in the weight space backpropagation (BP), better numerical conditioning, and lower computational cost compared to techniques based on the Hessian matrix. The numerical stability is assured by the use of robust LS linear system solvers, operating directly on the input data of each layer. Experimental results obtained in three problems are described, which confirm the effectiveness of the new method.

Recursive least squares (RLS)-based algorithms are a class of fast online training algorithms for feedforward multilayered neural networks (FMNNs). Though the standard RLS algorithm has an implicit weight decay term in its energy function, the weight decay effect decreases linearly as the number of learning epochs increases, thus rendering a diminishing weight decay effect as training progresses. In this paper, we derive two modified RLS algorithms to tackle this problem. In the first algorithm, namely, the true weight decay RLS (TWDRLS) algorithm, we consider a modified energy function whereby the weight decay effect remains constant, irrespective of the number of learning epochs. The second version, the input perturbation RLS (IPRLS) algorithm, is derived by requiring robustness in its prediction performance to input perturbations. Simulation results show that both algorithms improve the generalization capability of the trained network.

Network intrusion-detection systems are now being identified as a mandatory component in multilayered security architecture. Intrusion detection systems have traditionally been based on the characterization of a user and tracking of activity of the user to see if it matches that characterization. Artificial neural networks provide a feasible approach to model complex engineering systems such as intrusion detection. Applications of artificial neural networks to characterize the behavior of users have been well studied in the recent past, without considering the enormous time they take to get modeled. In this paper we present an implementation of a parallel version of the back propagation training algorithm for feed-forward neural networks that are used for detecting intruders based on the MPI standard on Linux PC clusters. The experiments show a considerable increase in speedup during training and testing of the neural network, which in turn increases the speed of detecting intruders.

In this paper, the terminal attractor based back propagation learning algorithms for feedforward neural networks are examined. Through a rigorous mathematical analysis, a condition to guarantee the convergence of the algorithms is given. A simulation study is presented to demonstrate the effectiveness of the analysis.

A comparison was carried out of several learning algorithms for training feedforward neural networks with linear threshold units. These learning algorithms do not require an a priori network architecture, but add neurons at will during training. The performance of these algorithms was compared by using training sets with a particular correlation of the input patterns over the full range of possible correlations. For binary input patterns we present a fast method for the selection of input patterns that can be identified by a single neuron. This method is not based on the perceptron learning rule.

In order to solve the problem that the traditional long-term high-speed traffic forecasting algorithm is affected by the approximation ability of the function and easy to fall into the local mass value, we wrote a multivariate-based highway traffic forecasting algorithm scaling and convolutional networks. Because the feedforward wavelet neural network algorithm predicts the short-term traffic flow in different areas, it is necessary to examine the ability to predict the difference between different models. From the standard feedforward wavelet neural network algorithm using global optimization capabilities, we improve the wolf pack algorithm, improve the search accuracy of the algorithm, get the best solution of the estimated value of the work according to the search results when completing the research objectives, and get the ability to predict the work of the model. Feedforward neural network algorithm: we develop and obtain the best short-term high-speed traffic forecast values. The results are as follows: after using the author’s algorithm, the processing time increases by 1.5 seconds, but the average percentage of errors decreases by more than 50%, in fact the error and the root mean square error decreased by about 30%, and the smoothing coefficient increased by about 1%. The prediction of the author’s algorithm for short-term high-speed traffic is better than the wavelet neural network prediction algorithm, and the prediction accuracy and stability of the author’s algorithm are higher.

Linear programming algorithm for neural networks

A fast and robust learning algorithm for feedforward neural networks

Error backpropagation neural network (EBP) used training algorithm for feedforward artificial neural networks (FFANNs). The main problem with the EBP algorithm that it is very slow and the converge to the optimal solution is not guaranteed. This problem leads to search for improvements to speed up this algorithm. In this research we use several methods to speed up the EBP algorithm. A many layer neural network was designed for building pattern compression system, encoding and recognition. We also used many methods to speed up this algorithm (EBP) and comparison between them.

Investigates linear separability in feature space from Tietze extension theorem and function approximation theory, and kernel technique is motivated theoretically. Two kernel-based algorithms are presented. One of them is a general learning algorithm for feedforward neural networks, and they can solve large-scale classification problems.

The convergence of a class of stochastic pseudo-gradient algorithms driven by correlated data sequences is considered in this paper. The obtained results are applied to a learning algorithm for feedforward neural networks and sufficient conditions for its convergence are determined.

Davidon least squares-based learning algorithm for feedforward neural networks