Fast Convergent Generalized Back-Propagation Algorithm with Constant Learning Rate

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

The conventional backpropagation algorithm is basically a local search technique which uses the gradient information for convergence. However, it has the problems of local minima and slow convergence. Evolution of the connection weights based on backpropagation can effectively avoid these local minima and speed up the convergence rate. The idea is to perturb the network weights in a controlled manner so as to 'jump off' from the local minima. In this paper, the weight evolution algorithm and the effect of parameters are thoroughly described. A mathematical analysis on the weight evolution algorithm is also included. Simulation results show that the weight evolution algorithm can effectively give fast learning behaviour with global search capability.

Stability analysis of a three-term backpropagation algorithm

Based on various approaches, several different learing algorithms have been given in the literature for neural networks. Almost all algorithms have constant learning rates or constant accelerative parameters, though they have been shown to be effective for some practical applications. The learning procedure of neural networks can be regarded as a problem of estimating (or identifying) constant parameters (i.e. connection weights of network) with a nonlinear or linear observation equation. Making use of the Kalman filtering, we derive a new back-propagation algorithm whose learning rate is computed by a time-varying Riccati difference equation. Perceptron-like and correlational learning algorithms are also obtained as special cases. Furthermore, a self-organising algorithm of feature maps is constructed within a similar framework.

Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

Against these disadvantages like local minimum, slow convergence speed, non-convergence and difficulty in obtaining of global optimal point, the new Neural Networks with Weight Function is proposed in this paper with simple network topology constituted by input layer and output layer only. This network is used in establishing the Energy Consumption Forecasting Model of DAGUSHAN Ore Dressing Plant. According to the production data in actual production process and the gap of these data, different interpolation functions are selected to be the weight functions. Simulation examples show the good performance of this method that little calculation work, with no local minimum and slow convergence problems. Model mentioned above has minor error and the better prediction effect is obtained.

The behavior of neural network learning algorithms with a small, constant learning rate, epsilon, in stationary, random input environments is investigated. It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as epsilon tends to zero. As applications, backpropagation in feedforward architectures and some feature extraction algorithms are studied in more detail.

To solve the slow convergence or non-convergence problem of traditional distance dynamics method, an improved community detection algorithm based on distance dynamics is proposed in this paper for speeding up the time efficient and accuracy of algorithm. Our improved algorithm firstly analyzes the node cohesiveness and slow convergence problems. Secondly, the neighbor cohesion is defined and two new interaction patterns are designed to enhance the accuracy of community. Finally, the convergence coefficient is defined to judge whether the dynamic interaction process is in the slow convergence or non-convergence status. In the dynamic interaction process, when the percentage of non-converged distance is less than convergence coefficient, the final value of the distances are pre-judged and the whole algorithm is over. The experimental results show our improved algorithm has a fast convergent speed and accurate detection results.

The back propagation (BP) algorithm is a very popular learning approach in feedforward multilayer perceptron networks. However, the most serious problem associated with the BP is local minima problem and slow convergence speeds. Over the years, many improvements and modifications of the back propagation learning algorithm have been reported. In this research, we propose a new modified back propagation learning algorithm by introducing adaptive gain together with adaptive momentum and adaptive learning rate into weight update process. By computer simulations, we demonstrate that the proposed algorithm can give a better convergence rate and can find a good solution in early time compare to the conventional back propagation. We use two common benchmark classification problems to illustrate the improvement in convergence time.Keywordsback propagationconvergence speedadaptive gainadaptive momentumadaptive learning rate

The cyclotron cavity presented in this paper is modeled by a feed-forward neural network trained by the authors’ optimized back-propagation (BP) algorithm. The training samples were obtained from simulation results that are for a number of defined situations and parameters and were achieved parametrically using MWS CST software; furthermore, the conventional BP algorithm with different hidden-neuron numbers, structures, and other optimal parameters such as learning rate that are applied for our purpose was also used here. The present study shows that an optimized FFN can be used to estimate the cyclotron-model parameters with an acceptable error function. A neural network trained by an optimized algorithm therefore shows a proper approximation and an acceptable ability regarding the modeling of the proposed structure. The cyclotron-cavity parameter-modeling results demonstrate that an FNN that is trained by the optimized algorithm could be a suitable method for the estimation of the design parameters in this case.

BP(Back Propagation) neural network, as a method of data fusion technology, has been used in many common fields widely. While, the main problem of BP algorithm is that the optimal procedure is easily trapped into local minimum value and the speed of convergence is very slow. To avoid this problem, this paper, which, making use of ergodicity property of chaos, starts its improvement from the learning rate. Validity of the proposed method is examined by performing simulations on network traffic prediction, the result shows that the improved algorithm not only is more efficient in internet traffic prediction with higher precision and faster speed of convergence, but also somewhat saves the network from the problem of local minima.

Inverted pendulums constitute one of the popular systems for benchmarking control algorithms. Several methods have been proposed for the control of this system, the majority of which rely on the availability of a mathematical model. However, deriving a mathematical model using physical parameters or system identification techniques requires manual effort. Moreover, the designed controllers may perform poorly if system parameters change. To mitigate these problems, recently, some studies used Reinforcement Learning (RL) based approaches for the control of inverted pendulum systems. Unfortunately, these methods suffer from slow convergence and local minimum problems. Moreover, they may require hyperparameter tuning which complicates the design process significantly. To alleviate these problems, the present study proposes an LQR-based RL method for adaptive balancing control of an inverted pendulum. As shown by numerical experiments, the algorithm stabilizes the system very fast without requiring a mathematical model or extensive hyperparameter tuning. In addition, it can adapt to parametric changes online.

With absorption and interference cancellation, lossy metamaterials can achieve broadband electromagnetic wave absorption. However, the design of such metamaterial absorbers (MMAs) is very complicate, because tremendous meta-atoms and their configuration parameters need to be determined. The conventional methods, such as parameter sweep and adjoint-based optimization, suffer from slow convergence and local minimum problem. In this letter, a deep neural network (DNN) is used to map the configuration parameters of a type of meta-atom onto its reflection coefficients. The DNN well predicts the reflection coefficients and is applied to design the MMA under the demand of −10 dB backscattering reduction (BSR) covering the microwave S to Ku bands with the smallest thickness. By globally searching the configuration parameter space using particle swarm optimization (PSO) algorithm, which automatically cooperates the absorption and phase cancelation (or diffusion) of meta-atoms, we obtain the optimized configuration of the meta-atoms and corresponding filling ratios. The designed MMA realizes −10 dB absorption bandwidth covering 2.2–18 GHz with the thickness only 4 mm, which is further verified by experiments. The performance of our absorber is better than other similar absorbers reported. Our letter provides a useful method for ultrathin broadband MMA design, which can also be applied to other functional devices based on metamaterials.

Back Propagation (BP) algorithm is one of the oldest learning techniques used by artificial neural networks (ANN). It has successfully been implemented in various practical problems. However, the algorithm still faces some drawbacks such as getting easily stuck at local minima and needs longer time to converge on an acceptable solution. Recently, the introduction of Second Order Methods has shown a significant improvement on the learning in BP but it still has some drawbacks such as slow convergence and complexity. To overcome these limitations, this research proposed a modified approach for BP by introducing the Conjugate Gradient and Quasi-Newton which were Second Order methods together with ‘gain’ parameter. The performances of the proposed approach is evaluated in terms of lowest number of epochs, lowest CPU time and highest accuracy on five benchmark classification datasets such as Glass, Horse, 7Bit Parity, Indian Liver Patient and Lung Cancer. The results show that the proposed Second Order methods with ‘gain’ performed better than the BP algorithm.

There are a number of problems associated with training neural networks with backpropagation algorithm. The algorithm scales exponentially with increased complexity of the problem. It is very often trapped in local minima, and is not robust to changes of network parameters such as number of hidden layer neurons and learning rate. The use of genetic algorithms is a recent trend, which is good at exploring a large and complex search space, to overcome such problems. In this paper a genetic algorithm is proposed for training feedforward neural networks and its performances is investigated. The results are analyzed and compared with those obtained by the backpropagation algorithm.