Electromagnetism-Like Mechanism Based Algorithm for Neural Network Training

The neural network (NN) training algorithm based on gradient optimization can not avoid falling into the local minimum because of the inappropriate initial weight value. The paper applies the genetic algorithm (GA) to training the linkage weights of NN. The training result can be used as the weights of an initial network for a back propagation (BP) training algorithm, then online optimization work can be done by BP training algorithm. It succeeds in avoiding the GA's defect of high calculating cost of every step, and giving full play to GA's advantage of greater probability of global convergence. Thus the online BP training algorithm can be lifted out of local minimum with greater probability and the better training property of the network is gained. The result of simulation shows that the robustness of the control system is improved.

There are numerous algorithms available for training artificial neural networks. Besides classical algorithms for supervised learning such as backpropagation, associative memory and radial basis function, this training task can be employed by evolutionary computation since most of the gradient descent related algorithms can be view as an application of optimization theory and stochastic search. In this paper, the logistic model of population growth from ecology is integrated into initialization, selection and crossover operators of genetic algorithms for neural network training. These chaotic operators are very efficient in maintaining the population diversity during the evolution process of genetic algorithms. A comparison is done on the basis of a benchmark comprising several data classification problems for neural networks. Three variants of training – Backpropagation (BP), Genetic Algorithms (GA) and Genetic Algorithms with Chaotic Operators (GACO) – are described and compared. The experimental results confirm the dynamic mobility of chaotic algorithms in GACO network training, which can overcome saturation and improve the convergence rate.KeywordsChaos theoryChaotic operatorEvolutionary algorithmsNeural networks training

In this paper, a meta-heuristic algorithm (electromagnetism-like mechanism, EM) for fuzzy neural network training is introduced. Electromagnetism-like mechanism simulates the electromagnetism theory of physics by considering each sample point to be an electrical charge. The EM algorithm utilizes an attraction-repulsion mechanism to move the sample points towards the optimum. Besides, the electromagnetism-like mechanism is not easily falling into local optimum. Therefore, the purpose of this study is to use the electromagnetism-like mechanism to develop the fuzzy neural networks (EMFNN), and employ this EMFNN to train fuzzy if-then rules. According to the case, the EMFNN could successfully generalize new fuzzy if-then rules.KeywordsLocal SearchFuzzy NumberFuzzy Neural NetworkFuzzy WeightSales ForecastThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Particle swarm optimization (PSO) motivated by the social behavior of organisms, is a step up to existing evolutionary algorithms for optimization of continuous nonlinear functions. Backpropagation (BP) is generally used for neural network training. Choosing a proper algorithm for training a neural network is very important. In this paper, a comparative study is made on the computational requirements of the PSO and BP as training algorithms for neural networks. Results are presented for a feedforward neural network learning a nonlinear function and these results show that the feedforward neural network weights converge faster with the PSO than with the BP algorithm.

Traditionally, the application of a neural network (Haykin, 1999) to solve a problem has required to follow some steps before to obtain the desired network. Some of these steps are the data preprocessing, model selection, topology optimization and then the training. It is usual to spend a large amount of computational time and human interaction to perform each task of before and, particularly, in the topology optimization and network training. There have been many proposals to reduce the effort necessary to do these tasks and to provide the experts with a robust methodology. For example, Giles et al. (1995) provides a constructive method to optimize iteratively the topology of a recurrent network. Other methods attempt to reduce the complexity of the network structure by mean of removing unnecessary network nodes and connections like in (Morse, 1994). In the last years, evolutionary algorithms have been shown as promising tools to solve this problem, existing many competitive approaches in the literature. For example, Blanco et al. (2001) proposed a master-slave genetic algorithm to train (master algorithm) and to optimize the size of the network (slave algorithm). For a general view of the problem and the use of evolutionary algorithms for neural network training and optimization, we refer the reader to (Yao, 1999). Although the literature about genetic algorithms and neural networks is very extensive, we would like to remark the recent popularity of multi-objective optimization (Coello et al., 2002, Jin, 2006), specially to solve the problem of simultaneous training and topology optimization of neural networks. These methods have shown to perform suitably for this task in previous works, although most of them are proposed for feedforward models. They attempt to optimize the structure of the network (number of connections, hidden units or layers), while training the network at the same time. Multi-objective algorithms may provide important advantages in the simultaneous training and optimization of neural networks: They may force the search to return a set of optimal networks instead of a single one; they are capable to speed-up the optimization process; they may be preferred to a weight-aggregation procedure to cover the regularization problem in neural networks; and they are more suitable when the designer would like to combine different error measures for the training. A recent review of these techniques may be found in (Jin, 2006).

This paper presents a new approach based on a Two-Stage Linear Discriminant Analysis (Two-Stage LDA) and Conjugate Gradient Algorithms (CGAs) for face recognition. A Two-Stage LDA technique is proposed that utilises the null space of the sample covariance matrix as well as using the range space of the between-class scatter matrix to extract discriminant information. Classic Back Propagation (BP) is a widely used Neural Network (NN) training algorithm in many detectors and classifiers. However, it is both too slow for many practical problems and its performance is not satisfactory in many application areas, including face recognition. To overcome these problems, four CGA algorithms (Fletcher-Reeves CGA, Polak-Ribiere CGA, Powell-Beale CGA, scaled CGA) have been proposed, the utility of which we investigate here in combination with Two-Stage LDA features. To further improve the accuracy, a modified AdaBoost.M1 approach was employed, which combines results of several NN classifiers as a single strong classifier. Experiments are performed on the ORL, FERET and AR face databases. The results show that all of the proposed methods lead to increased recognition rates and shorter training times compared to the classic BP.

Machine learning approaches have been promising in constructing high-dimensional potential energy surfaces (PESs) for molecules and materials. Neural networks (NNs) are one of the most popular such tools because of its simplicity and efficiency. The training algorithm for NNs becomes essential to achieve a fast and accurate fit with numerous data. The Levenberg-Marquardt (LM) algorithm has been recognized as one of the fastest and robust algorithms to train medium sized NNs and widely applied in recent NN based high quality PESs. However, when the number of ab initio data becomes large, the efficiency of LM is limited, making the training time consuming. Extreme learning machine (ELM) is a recently proposed algorithm which determines the weights and biases of a single hidden layer NN by a linear solution and is thus extremely fast. It, however, does not produce sufficiently small fitting error because of its random nature. Taking advantages of both algorithms, we report a generalized hybrid algorithm in training multilayer NNs. Tests on H+H2 and CH4+Ni(111) systems demonstrate the much higher efficiency of this hybrid algorithm (ELM-LM) over the original LM. We expect that ELM-LM will find its widespread applications in building up high-dimensional NN based PESs.

The sensitivity of the outputs of a neural network to perturbations in its weights is an important consideration in both the design of hardware realizations and in the development of training algorithms for neural networks. In designing dense, high-speed realizations of neural networks, understanding the consequences of using simple neurons with significant weight errors is important. Similarly, in developing training algorithms, it is important to understand the effects of small weight changes to determine the required precision of the weight updates at each iteration. In this paper, an analysis of the sensitivity of feedforward neural networks (Madalines) to weight errors is considered. We focus our attention on Madalines composed of sigmoidal, threshold, and linear units. Using a stochastic model for weight errors, we derive simple analytical expressions for the variance of the output error of a Madaline. These analytical expressions agree closely with simulation results. In addition, we develop a technique for selecting the appropriate accuracy of the weights in a neural network realization. Using this technique, we compare the required weight precision for threshold versus sigmoidal Madalines. We show that for a given desired variance of the output error, the weights of a threshold Madaline must be more accurate.

Stochastic gradient descent is the most prevalent algorithm to train neural networks. However, other approaches such as evolutionary algorithms are also applicable to this task. Evolutionary algorithms bring unique trade-offs that are worth exploring, but computational demands have so far restricted exploration to small networks with few parameters. We implement an evolutionary algorithm that executes entirely on the GPU, which allows to efficiently batch-evaluate a whole population of networks. Within this framework, we explore the limited evaluation evolutionary algorithm for neural network training and find that its batch evaluation idea comes with a large accuracy trade-off. In further experiments, we explore crossover operators and find that unprincipled random uniform crossover performs extremely well. Finally, we train a network with 92k parameters on MNIST using an EA and achieve 97.6% test accuracy compared to 98% test accuracy on the same network trained with Adam. Code is available at https://github.com/jprellberg/gpuea.

There is a growing consensus that 3D printing technologies will be one of the next major technological revolutions. Over the past few years, many studies and researches have been carried out to improve and enhance the functions and performance of the 3D printers. In this paper, a modified Electromagnetism-like Mechanism (EM) algorithm is proposed to search for the optimum printing path of a 3D printer. The machine is set to fabricate the bracket of a photovoltaic solar energy harvesting panel. Instead of randomly setting the search step size, the improved EM systematically fine-tunes the steps to search for the best printing path. The performance of the enhanced EM is benchmarked with the conventional EM and Genetic Algorithm (GA). The results presented in this paper shows that the modified EM outperformed all other optimization techniques in terms of time taken, distance traveled and overall convergence process. We can thus conclude that the proposed modified EM performs well in optimizing the path planning sequence of a 3D printer.

This paper compares known architectures and training algorithms of multi-layered artificial neural networks (ANNs) and their ability to solve one specific type of chess problem (the Hamiltonian Cycles known as the Knight's Tours). The networks used here are trained to solve tours starting from the center of a five-by- five chessboard. The ANN architectures focused on are the function approximation feedforward architecture; the classification feedforward architecture; and two customized variations of the function approximation feedforward architecture. The training algorithms compared include Levenberg-Marquardt; variable learning (adaptive) gradient descent; resilient backpropagation; the Fletcher-Reeves conjugate gradient; and the Broyden-Fletcher-Goldfarb-Shanno Quasi-Newton algorithm. These five training algorithms are used for each of the four architectures. The real world application of this research is limited only by the functions given and the parameters used to define the environment. This can be expanded to include a variety of chess pieces (functions) in a variety of environments (any m-by-n sized board or even multidimensional problem solving). The practical limitation on the application is the software necessary to run extremely large neural networks.

The aim of the project is to study, analyze and research different models of algorithms for neural network training and the application of machine learning. The main goal is to offer a solution that detects malicious files in a large set of data by evaluating each of them. The algorithms are considered Gradient Boosting Classifier, Random Forest Classifier, also, in the proposed appropriate solutions to this problem, neural networks with „deep learning”, as well as techniques from hidden Markov model. They are often a good solution for detecting and classifying malware.

Backpropagation is the most popular algorithm for training neural networks. However, this gradient-based training method is known to have a tendency towards very long training times and convergence to local optima. Various methods have been proposed to alleviate these issues including, but not limited to, different training algorithms, automatic architecture design and different transfer functions. In this chapter we continue the exploration into improving gradient-based learning algorithms through dynamic transfer function modification. We propose opposite transfer functions as a means to improve the numerical conditioning of neural networks and extrapolate two backpropagation-based learning algorithms. Our experimental results show an improvement in accuracy and generalization ability on common benchmark functions. The experiments involve examining the sensitivity of the approach to learning parameters, type of transfer function and number of neurons in the network.

The performance of three recently presented training algorithms in neural networks is investigated. These algorithms are robust to infeasible problems, in which case an appropriate error function is minimized. In the infeasibility regime, simulations are performed and compared to recently published analytical work in one-step replica symmetry broken theory. A careful analysis explains insufficiencies in these analytic results. A new stability result in the infeasibility regime is derived and shown to match simulation data.

We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).