Performance of robust training algorithms for neural networks

Particle swarm optimization (PSO) motivated by the social behavior of organisms, is a step up to existing evolutionary algorithms for optimization of continuous nonlinear functions. Backpropagation (BP) is generally used for neural network training. Choosing a proper algorithm for training a neural network is very important. In this paper, a comparative study is made on the computational requirements of the PSO and BP as training algorithms for neural networks. Results are presented for a feedforward neural network learning a nonlinear function and these results show that the feedforward neural network weights converge faster with the PSO than with the BP algorithm.

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

We propose an evolutionary neural network-training algorithm for beta basis function neural networks (BBFNN). Classic training algorithms for neural networks start with a predetermined network structure. Generally the network resulting from learning applied to a predetermined architecture is either insufficient or over-complicated. This paper describes a hierarchical genetic learning model of the BBFNN. In order to examine the performance of the proposed algorithm, they were used for the approximation problems. The results obtained are very satisfactory with respect to the relative error.

The sensitivity of the outputs of a neural network to perturbations in its weights is an important consideration in both the design of hardware realizations and in the development of training algorithms for neural networks. In designing dense, high-speed realizations of neural networks, understanding the consequences of using simple neurons with significant weight errors is important. Similarly, in developing training algorithms, it is important to understand the effects of small weight changes to determine the required precision of the weight updates at each iteration. In this paper, an analysis of the sensitivity of feedforward neural networks (Madalines) to weight errors is considered. We focus our attention on Madalines composed of sigmoidal, threshold, and linear units. Using a stochastic model for weight errors, we derive simple analytical expressions for the variance of the output error of a Madaline. These analytical expressions agree closely with simulation results. In addition, we develop a technique for selecting the appropriate accuracy of the weights in a neural network realization. Using this technique, we compare the required weight precision for threshold versus sigmoidal Madalines. We show that for a given desired variance of the output error, the weights of a threshold Madaline must be more accurate.

This paper reports the performance of a novel training algorithm for quantum neural networks (QNN) using a variation of the quantum search algorithm. The proposed algorithm trains a QNN exploring all possible weights with a sublinear cost. The training cost is theoretically O(√N/t), as a function of the quantity N of possible weights and t is the number of possible solutions. Initial experimental results demonstrate that the algorithm always converges to existing solutions, in addition to having the mean and maximum values, almost in total, lower than the expected maximum amount, as theoretically expected. The training algorithm is applied to classification problems.

This paper explores feasibility of employing the non-recurrent backpropagation training algorithm for a recurrent neural network, Simultaneous Recurrent Neural network, for static optimisation. A simplifying observation that maps the recurrent network dynamics, which is configured to operate in relaxation mode as a static optimizer, to feedforward network dynamics is leveraged to facilitate application of a non-recurrent training algorithm such as the standard backpropagation and its variants. A simulation study that aims to assess feasibility, optimizing potential, and computational efficiency of training the Simultaneous Recurrent Neural network with non-recurrent backpropagation is conducted. A comparative computational complexity analysis between the Simultaneous Recurrent Neural network trained with non-recurrent backpropagation algorithm and the same network trained with the recurrent backpropagation algorithm is performed. Simulation results demonstrate that it is feasible to apply the non-recurrent backpropagation to train the Simultaneous Recurrent Neural network. The optimality and computational complexity analysis fails to demonstrate any advantage on behalf of the non-recurrent backpropagation versus the recurrent backpropagation for the optimisation problem considered. However, considerable future potential that is yet to be explored exists given that computationally efficient versions of the backpropagation training algorithm, namely quasi-Newton and conjugate gradient descent among others, are also applicable for the neural network proposed for static optimisation in this paper.

We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

A new fast learning algorithm for a multilayer feedforward neural network is presented. The new algorithm, based on an innovative variable step size and gradient averaging, has excellent properties to overcome major drawbacks of the backpropagation algorithm and shows high convergence speed. The improved performance of the new algorithm in comparison with other algorithms is verified via computer simulations.

In recent years many training algorithms for dynamic neural networks have been proposed. As a matter of fact, it is well known that the exact training algorithm for dynamic networks, is non causal and can be implemented only in batch mode. In this paper we present a comparison of three online training algorithms for dynamic networks, where each synapsis is modelled by an FIR filter. In order to evaluate performance and computational complexity of the various algorithms, several computer simulations of dynamical system identifications have been carried out.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A fast training algorithm for artificial neural networks using a feedforward multilayer perceptron architecture is presented. The application of this algorithm to the problem of multiuser detection in the synchronous DS CDMA channel is investigated. The performance of this multiuser detector is shown to be very close to the single user bound. The training algorithm is compared with the conventional gradient descent-backpropagation algorithm. It is demonstrated that the new training algorithm converges significantly faster than the backpropagation algorithm, while keeping the performance constant. Finally, a pre-training algorithm is proposed in order to further reduce the length of the required training sequence.

This paper compares known architectures and training algorithms of multi-layered artificial neural networks (ANNs) and their ability to solve one specific type of chess problem (the Hamiltonian Cycles known as the Knight's Tours). The networks used here are trained to solve tours starting from the center of a five-by- five chessboard. The ANN architectures focused on are the function approximation feedforward architecture; the classification feedforward architecture; and two customized variations of the function approximation feedforward architecture. The training algorithms compared include Levenberg-Marquardt; variable learning (adaptive) gradient descent; resilient backpropagation; the Fletcher-Reeves conjugate gradient; and the Broyden-Fletcher-Goldfarb-Shanno Quasi-Newton algorithm. These five training algorithms are used for each of the four architectures. The real world application of this research is limited only by the functions given and the parameters used to define the environment. This can be expanded to include a variety of chess pieces (functions) in a variety of environments (any m-by-n sized board or even multidimensional problem solving). The practical limitation on the application is the software necessary to run extremely large neural networks.

Network intrusion-detection systems are now being identified as a mandatory component in multilayered security architecture. Intrusion detection systems have traditionally been based on the characterization of a user and tracking of activity of the user to see if it matches that characterization. Artificial neural networks provide a feasible approach to model complex engineering systems such as intrusion detection. Applications of artificial neural networks to characterize the behavior of users have been well studied in the recent past, without considering the enormous time they take to get modeled. In this paper we present an implementation of a parallel version of the back propagation training algorithm for feed-forward neural networks that are used for detecting intruders based on the MPI standard on Linux PC clusters. The experiments show a considerable increase in speedup during training and testing of the neural network, which in turn increases the speed of detecting intruders.

The problems of neural network algorithms designing for modeling of the switching technical systems are considered. The method for designing of dynamic models using polynomial approximation is proposed. The generalized models of switching systems taking into account nonstationary perturbations are constructed. The heuristic optimization algorithms implemented in the form of computer libraries are developed. The problems of neural network algorithm implementation using high-level hybrid computing are considered. The effect of the application of the algorithms proposed in the paper is to reduce the time and energy costs on creating vector thrust. Algorithmic support based on artificial neural networks with training is proposed. The library of high-parallel training algorithms for neural networks in the problems of optimal trajectories constructing for switching technical systems is developed. The obtained results can be used in problems of researching models of controlled technical systems with switching operating modes, in particular, in modeling the dynamics of various classes of aircraft and in modeling intelligent transport systems #CSOC1120.

Classic training algorithms for neural networks start with a predetermined network structure, and so the quality of the response of the neural network depends strongly on its structure. Generally the neural network resulting from such classical learning approach applied to a predetermined architecture is either insufficient or overcomplicated. This paper describes two genetic learning models of the BBFNN. The first is a continuous genetic and the second is a discrete genetic model. In the two cases each network is coded as a variable length string and some genetic operators are proposed to evolve a population of individuals. A function is proposed to evaluate the fitness of individual networks. Applications to function approximation problems are considered to demonstrate the performance of the BBFNN and of the two evolutionary algorithms.

In this work, a new training algorithm for probabilistic neural networks (PNN) is presented. The proposed algorithm addresses one of the major drawbacks of probabilistic neural networks, which is the size of the hidden layer in the network. By using a cross-validation training algorithm, the number of hidden neurons is shrunk to a smaller number consisting of the most representative samples of the training set. This is done without affecting the overall architecture of the network. Performance of the new network is compared against performance of standard probabilistic neural networks for different databases from the UCI database repository. Results show an important gain in network size and performance.