Neural Network Optimization Algorithms for Controlled Switching Systems

Particle swarm optimization (PSO) motivated by the social behavior of organisms, is a step up to existing evolutionary algorithms for optimization of continuous nonlinear functions. Backpropagation (BP) is generally used for neural network training. Choosing a proper algorithm for training a neural network is very important. In this paper, a comparative study is made on the computational requirements of the PSO and BP as training algorithms for neural networks. Results are presented for a feedforward neural network learning a nonlinear function and these results show that the feedforward neural network weights converge faster with the PSO than with the BP algorithm.

This paper compares known architectures and training algorithms of multi-layered artificial neural networks (ANNs) and their ability to solve one specific type of chess problem (the Hamiltonian Cycles known as the Knight's Tours). The networks used here are trained to solve tours starting from the center of a five-by- five chessboard. The ANN architectures focused on are the function approximation feedforward architecture; the classification feedforward architecture; and two customized variations of the function approximation feedforward architecture. The training algorithms compared include Levenberg-Marquardt; variable learning (adaptive) gradient descent; resilient backpropagation; the Fletcher-Reeves conjugate gradient; and the Broyden-Fletcher-Goldfarb-Shanno Quasi-Newton algorithm. These five training algorithms are used for each of the four architectures. The real world application of this research is limited only by the functions given and the parameters used to define the environment. This can be expanded to include a variety of chess pieces (functions) in a variety of environments (any m-by-n sized board or even multidimensional problem solving). The practical limitation on the application is the software necessary to run extremely large neural networks.

Network intrusion-detection systems are now being identified as a mandatory component in multilayered security architecture. Intrusion detection systems have traditionally been based on the characterization of a user and tracking of activity of the user to see if it matches that characterization. Artificial neural networks provide a feasible approach to model complex engineering systems such as intrusion detection. Applications of artificial neural networks to characterize the behavior of users have been well studied in the recent past, without considering the enormous time they take to get modeled. In this paper we present an implementation of a parallel version of the back propagation training algorithm for feed-forward neural networks that are used for detecting intruders based on the MPI standard on Linux PC clusters. The experiments show a considerable increase in speedup during training and testing of the neural network, which in turn increases the speed of detecting intruders.

Pattern classification via linear programming

We propose an evolutionary neural network-training algorithm for beta basis function neural networks (BBFNN). Classic training algorithms for neural networks start with a predetermined network structure. Generally the network resulting from learning applied to a predetermined architecture is either insufficient or over-complicated. This paper describes a hierarchical genetic learning model of the BBFNN. In order to examine the performance of the proposed algorithm, they were used for the approximation problems. The results obtained are very satisfactory with respect to the relative error.

Artificial neural networks are inherently high-dimensional, which limits our ability to visualise and understand their inner workings. Neural network architecture and training algorithm parameters are usually optimised on an ad hoc basis, with very limited insight into the nature of the objective function landscape. This study proposes using fitness landscape analysis to quantify topological properties of neural network error landscapes. Five techniques from the fitness landscape analysis field are adapted to work with neural network error landscapes. These techniques are then used to analyse how the error landscape changes under different error measurements and different number of hidden layers. The results show that fitness landscape analysis provides valuable insight into neural network error landscapes, and could be used for architecture selection.

There are numerous algorithms available for training artificial neural networks. Besides classical algorithms for supervised learning such as backpropagation, associative memory and radial basis function, this training task can be employed by evolutionary computation since most of the gradient descent related algorithms can be view as an application of optimization theory and stochastic search. In this paper, the logistic model of population growth from ecology is integrated into initialization, selection and crossover operators of genetic algorithms for neural network training. These chaotic operators are very efficient in maintaining the population diversity during the evolution process of genetic algorithms. A comparison is done on the basis of a benchmark comprising several data classification problems for neural networks. Three variants of training – Backpropagation (BP), Genetic Algorithms (GA) and Genetic Algorithms with Chaotic Operators (GACO) – are described and compared. The experimental results confirm the dynamic mobility of chaotic algorithms in GACO network training, which can overcome saturation and improve the convergence rate.KeywordsChaos theoryChaotic operatorEvolutionary algorithmsNeural networks training

"This paper uses an artificial neural network (ANN) and resilient back-propagation (Rprop) training algorithm to determine the optical constants of As 30 Se 70 − x Sn x (0 ≤ x ≤ 3) thin films. The simulated values of the ANN are in good agreement with the experimental data. The ANN models performance was also examined to predict the simulated values for As 30 Se 67 Sn 3 which was not included in the training and was found to be in accordance with the experimental data. The high precision of the ANN models as well as a great guessing performance have been exhibited. Moreover, the energy gap E g of As 30 Se 70 − x Sn x (0 ≤ x ≤ 9) thin films were calculated theoretically."

Classic training algorithms for neural networks start with a predetermined network structure, and so the quality of the response of the neural network depends strongly on its structure. Generally the neural network resulting from such classical learning approach applied to a predetermined architecture is either insufficient or overcomplicated. This paper describes two genetic learning models of the BBFNN. The first is a continuous genetic and the second is a discrete genetic model. In the two cases each network is coded as a variable length string and some genetic operators are proposed to evolve a population of individuals. A function is proposed to evaluate the fitness of individual networks. Applications to function approximation problems are considered to demonstrate the performance of the BBFNN and of the two evolutionary algorithms.

Traditionally, the application of a neural network (Haykin, 1999) to solve a problem has required to follow some steps before to obtain the desired network. Some of these steps are the data preprocessing, model selection, topology optimization and then the training. It is usual to spend a large amount of computational time and human interaction to perform each task of before and, particularly, in the topology optimization and network training. There have been many proposals to reduce the effort necessary to do these tasks and to provide the experts with a robust methodology. For example, Giles et al. (1995) provides a constructive method to optimize iteratively the topology of a recurrent network. Other methods attempt to reduce the complexity of the network structure by mean of removing unnecessary network nodes and connections like in (Morse, 1994). In the last years, evolutionary algorithms have been shown as promising tools to solve this problem, existing many competitive approaches in the literature. For example, Blanco et al. (2001) proposed a master-slave genetic algorithm to train (master algorithm) and to optimize the size of the network (slave algorithm). For a general view of the problem and the use of evolutionary algorithms for neural network training and optimization, we refer the reader to (Yao, 1999). Although the literature about genetic algorithms and neural networks is very extensive, we would like to remark the recent popularity of multi-objective optimization (Coello et al., 2002, Jin, 2006), specially to solve the problem of simultaneous training and topology optimization of neural networks. These methods have shown to perform suitably for this task in previous works, although most of them are proposed for feedforward models. They attempt to optimize the structure of the network (number of connections, hidden units or layers), while training the network at the same time. Multi-objective algorithms may provide important advantages in the simultaneous training and optimization of neural networks: They may force the search to return a set of optimal networks instead of a single one; they are capable to speed-up the optimization process; they may be preferred to a weight-aggregation procedure to cover the regularization problem in neural networks; and they are more suitable when the designer would like to combine different error measures for the training. A recent review of these techniques may be found in (Jin, 2006).

The sensitivity of the outputs of a neural network to perturbations in its weights is an important consideration in both the design of hardware realizations and in the development of training algorithms for neural networks. In designing dense, high-speed realizations of neural networks, understanding the consequences of using simple neurons with significant weight errors is important. Similarly, in developing training algorithms, it is important to understand the effects of small weight changes to determine the required precision of the weight updates at each iteration. In this paper, an analysis of the sensitivity of feedforward neural networks (Madalines) to weight errors is considered. We focus our attention on Madalines composed of sigmoidal, threshold, and linear units. Using a stochastic model for weight errors, we derive simple analytical expressions for the variance of the output error of a Madaline. These analytical expressions agree closely with simulation results. In addition, we develop a technique for selecting the appropriate accuracy of the weights in a neural network realization. Using this technique, we compare the required weight precision for threshold versus sigmoidal Madalines. We show that for a given desired variance of the output error, the weights of a threshold Madaline must be more accurate.

Computer software is frequently used for medical decision support systems in different areas. Magnetic Resonance Images (MRI) are widely used images for brain classification issue. This paper presents an improved method for brain classification of MRI images. The proposed method contains three phases, which are, feature extraction, dimensionality reduction, and an improved classification technique. In the first phase, the features of MRI images are obtained by discrete wavelet transform (DWT). In the second phase, the features of MRI images have been reduced, using principal component analysis (PCA). In the last (third) stage, an improved classifier is developed. In the proposed classifier, Dragonfly algorithm is used instead of backpropagation as training algorithm for artificial neural network (ANN). Some other recent training-based Neural Networks, SVM, and KNN classifiers are used for comparison with the proposed classifier. The classifiers are utilized to classify image as normal or abnormal MRI human brain image. The results show that the proposed classifier is outperformed the other competing classifiers.

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

The Dragonfly Algorithm (DA), a swarm intelligence algorithm inspired by the behavior of dragonflies, has been recently proposed and it was found to have a higher performance as compared to other swarm intelligence algorithms. However, it still has certain limitations, and its performance can be further improved. DA has a low exploitation phase, and this leads to problems such as low accuracy of solutions, falling in local optima and low convergence rate. We propose to enhance the exploitation of DA by using the hill climbing algorithm as a local search so as to increase its effectiveness and efficiency in producing high accuracy solutions. The hill climbing algorithm is chosen to be used as a local search for DA since it always optimizes the current solution until the local optima is obtained However, it has not been employed in any existing hybrid of DA. The proposed algorithm performance is to be evaluated by employing it as a training algorithm for an Artificial Neural Network (ANN) to optimize its connection weights. A classification dataset will be used for the training and testing of the ANN trained by the proposed algorithm. The root mean squared error of the ANN will be taken as the objective function of the optimized dragonfly algorithm. The accuracy of the resultant neural network will be compared to that of an ANN trained by the original DA and the time taken for the training process by both algorithms will be compared. Based on the analysis, the optimized algorithm is expected to outperform the original DA by allowing the resultant neural network to have a higher classification accuracy.

This paper presents an intelligent means of addressing characterization and grading problems in the oil palm industry for the purpose of quality control. A Layer-Sensitivity Based Artificial Neural Network (LSB_ANN) which updates its layer weights based on sensitivity analysis was designed to predict the oil content and dielectric constant of mature oil palm fruitlets. The LSB_ANN was designed, optimized and trained with 604 data points obtained from laboratory microwave coaxial sensor measurements within 2-4 GHz. The performance evaluation of the model when tested with a separate set of data showed that the properties of the fruitlets were accurately modeled. To further investigate the generalization ability of the trained neural network, three other neural network training algorithms were deployed for the same dataset. A multi-criteria evaluation of the performances of the networks showed that the proposed LSB_ANN outperformed the other three in generalization accuracy, time and computing resources. The LSB_ANN therefore represents a handy tool for rapid and intelligent characterization of oil palm fruitlets for quality control and research purposes.