Neighborhood based Levenberg-Marquardt algorithm for neural network training

In this paper we introduce an advanced supervised training method for neural networks. It is based on Jacobian rank deficiency and it is formulated, in some sense, in the spirit of the Gauss-Newton algorithm. The Levenberg-Marquardt algorithm, as a modified Gauss-Newton, has been used successfully in solving nonlinear least squares problems including neural-network training. It outperforms (in terms of training accuracy, convergence properties, overall training time, etc.) the basic backpropagation and its variations with variable learning rate significantly, however, with higher computation and memory complexities within each iteration. The new method developed in this paper is aiming at improving convergence properties, while reducing the memory and computation complexities in supervised training of neural networks. Extensive simulation results are provided to demonstrate the superior performance of the new algorithm over the Levenberg-Marquardt algorithm.

Traditionally, the application of a neural network (Haykin, 1999) to solve a problem has required to follow some steps before to obtain the desired network. Some of these steps are the data preprocessing, model selection, topology optimization and then the training. It is usual to spend a large amount of computational time and human interaction to perform each task of before and, particularly, in the topology optimization and network training. There have been many proposals to reduce the effort necessary to do these tasks and to provide the experts with a robust methodology. For example, Giles et al. (1995) provides a constructive method to optimize iteratively the topology of a recurrent network. Other methods attempt to reduce the complexity of the network structure by mean of removing unnecessary network nodes and connections like in (Morse, 1994). In the last years, evolutionary algorithms have been shown as promising tools to solve this problem, existing many competitive approaches in the literature. For example, Blanco et al. (2001) proposed a master-slave genetic algorithm to train (master algorithm) and to optimize the size of the network (slave algorithm). For a general view of the problem and the use of evolutionary algorithms for neural network training and optimization, we refer the reader to (Yao, 1999). Although the literature about genetic algorithms and neural networks is very extensive, we would like to remark the recent popularity of multi-objective optimization (Coello et al., 2002, Jin, 2006), specially to solve the problem of simultaneous training and topology optimization of neural networks. These methods have shown to perform suitably for this task in previous works, although most of them are proposed for feedforward models. They attempt to optimize the structure of the network (number of connections, hidden units or layers), while training the network at the same time. Multi-objective algorithms may provide important advantages in the simultaneous training and optimization of neural networks: They may force the search to return a set of optimal networks instead of a single one; they are capable to speed-up the optimization process; they may be preferred to a weight-aggregation procedure to cover the regularization problem in neural networks; and they are more suitable when the designer would like to combine different error measures for the training. A recent review of these techniques may be found in (Jin, 2006).

The Levenberg-Marquardt and Newton are two algorithms that use the Hessian for the artificial neural network learning. In this article, we propose a modified Levenberg-Marquardt algorithm for the artificial neural network learning containing the training and testing stages. The modified Levenberg-Marquardt algorithm is based on the Levenberg-Marquardt and Newton algorithms but with the following two differences to assure the error stability and weights boundedness: 1) there is a singularity point in the learning rates of the Levenberg-Marquardt and Newton algorithms, while there is not a singularity point in the learning rate of the modified Levenberg-Marquardt algorithm and 2) the Levenberg-Marquardt and Newton algorithms have three different learning rates, while the modified Levenberg-Marquardt algorithm only has one learning rate. The error stability and weights boundedness of the modified Levenberg-Marquardt algorithm are assured based on the Lyapunov technique. We compare the artificial neural network learning with the modified Levenberg-Marquardt, Levenberg-Marquardt, Newton, and stable gradient algorithms for the learning of the electric and brain signals data set.

A neural network training algorithm utilizing multiple sets of linear equations

Machine learning approaches have been promising in constructing high-dimensional potential energy surfaces (PESs) for molecules and materials. Neural networks (NNs) are one of the most popular such tools because of its simplicity and efficiency. The training algorithm for NNs becomes essential to achieve a fast and accurate fit with numerous data. The Levenberg-Marquardt (LM) algorithm has been recognized as one of the fastest and robust algorithms to train medium sized NNs and widely applied in recent NN based high quality PESs. However, when the number of ab initio data becomes large, the efficiency of LM is limited, making the training time consuming. Extreme learning machine (ELM) is a recently proposed algorithm which determines the weights and biases of a single hidden layer NN by a linear solution and is thus extremely fast. It, however, does not produce sufficiently small fitting error because of its random nature. Taking advantages of both algorithms, we report a generalized hybrid algorithm in training multilayer NNs. Tests on H+H2 and CH4+Ni(111) systems demonstrate the much higher efficiency of this hybrid algorithm (ELM-LM) over the original LM. We expect that ELM-LM will find its widespread applications in building up high-dimensional NN based PESs.

A Cardiovascular disease which is additionally referred to as heart diseases that have been a common and steady issue in the field of medical research. Nowadays various methods were applied which is not robust for the prediction of human being expenses and disease risks for patients. This paper proposed an Orthogonal Local Preserving Projection (OLPP) method to reduce the function dimension of the input high-dimensional data. The dimension reduction improves the prediction rate with the help of hybrid classifier i.e. Group Search Optimization Algorithm (GSO) combine with the Levenberg-Marquardt (LM) training algorithm in the neural network. The LM training algorithm is used to solve the optimization problem and it determines the best network parameters such as weights and bias that minimizes the error. The final output of the optimization technique is combined with the performance metrics as accuracy, sensitivity, and specificity. From the result, it is observed that hybrid optimization techniques increase the accuracy of the heart disease prediction system.

There are numerous algorithms available for training artificial neural networks. Besides classical algorithms for supervised learning such as backpropagation, associative memory and radial basis function, this training task can be employed by evolutionary computation since most of the gradient descent related algorithms can be view as an application of optimization theory and stochastic search. In this paper, the logistic model of population growth from ecology is integrated into initialization, selection and crossover operators of genetic algorithms for neural network training. These chaotic operators are very efficient in maintaining the population diversity during the evolution process of genetic algorithms. A comparison is done on the basis of a benchmark comprising several data classification problems for neural networks. Three variants of training – Backpropagation (BP), Genetic Algorithms (GA) and Genetic Algorithms with Chaotic Operators (GACO) – are described and compared. The experimental results confirm the dynamic mobility of chaotic algorithms in GACO network training, which can overcome saturation and improve the convergence rate.KeywordsChaos theoryChaotic operatorEvolutionary algorithmsNeural networks training

Due to the complex nature of training neural network (NN), this problem has gained popularity in the nonlinear optimization field. In order to avoid falling into local minimum because of inappropriate initial weights, a number of global search techniques are developed. This paper applies a novel global algorithm, which is electromagnetism-like mechanism (EM) algorithm, to train NN and the EM based algorithm for neural network training is presented. The performance of the proposed algorithm is evaluated in classification problems and the comparison with BP and GA algorithms shows its effectiveness.

Artificial neural networks modelling: Gasification behaviour of palm fibre biochar

The prediction of stock index movement is considered a rather significant objective in the financial world, since a reasonably accurate prediction has the possibility of gaining profit in stock exchange, yielding high financial benefits and hedging against market risks. Undoubtedly, the area of financial analysis has been dramatically changed from a rather qualitative science to a more quantitative science which is also based on knowledge extraction from databases. During the last years, deep learning constitutes a significant prediction tool in analyzing and exploiting the knowledge acquired from financial data. In this paper, we propose a new Deep Neural Network (DNN) prediction model for forecasting stock exchange index movement. The proposed DNN is characterized by the application of conditions on the weights in the form of box-constraints, during the training process. The motivation for placing these constraints is focused on defining the weights in the trained network in more uniform way, by restricting them from taking large values in order for all inputs and neurons of the DNN to be efficiently exploited and explored. The training of the new DNN model is performed by a Weight-Constrained Deep Neural Network (WCDNN) training algorithm which exploits the numerical efficiency and very low memory requirements of the L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) matrices together with a gradient-projection strategy for handling the bounds on the weights of the network. The performance evaluation carried out on three popular stock exchange indices, demonstrates the classification efficiency of the proposed algorithm.

Financial time series prediction using artificial neural network based on Levenberg-Marquardt algorithm

This paper describes an application of the structured genetic algorithm to the construction of microwave circuit neural network models. The algorithm is used to create a neural network model of a high electron mobility transistor (HEMT) power amplifier. Results are compared to other neural network training algorithms such as back-propagation and the Levenberg-Marquardt algorithm. Models of one and two-stage amplifiers created with the ADS circuit simulator give good predictions of measurement results.

The traditional approaches are insufficient to solve the complex kinematics problems of the redundant robotic manipulators. To overcome such intricacy, ANNs are used nowadays. The performance of the neural network is affected by the training algorithm and network topology. There are numerous training algorithms which are used in the training of neural networks. In this paper, Levenberg-Marquardt is used in training algorithm and its effect on the performance of the neural network on the inverse kinematics model learning of a 6-R robot is studied.

Speed and accuracy investigations of neural network algorithms for ionospheric modelling at an equatorial region

This paper compares known architectures and training algorithms of multi-layered artificial neural networks (ANNs) and their ability to solve one specific type of chess problem (the Hamiltonian Cycles known as the Knight's Tours). The networks used here are trained to solve tours starting from the center of a five-by- five chessboard. The ANN architectures focused on are the function approximation feedforward architecture; the classification feedforward architecture; and two customized variations of the function approximation feedforward architecture. The training algorithms compared include Levenberg-Marquardt; variable learning (adaptive) gradient descent; resilient backpropagation; the Fletcher-Reeves conjugate gradient; and the Broyden-Fletcher-Goldfarb-Shanno Quasi-Newton algorithm. These five training algorithms are used for each of the four architectures. The real world application of this research is limited only by the functions given and the parameters used to define the environment. This can be expanded to include a variety of chess pieces (functions) in a variety of environments (any m-by-n sized board or even multidimensional problem solving). The practical limitation on the application is the software necessary to run extremely large neural networks.