Stability analysis of a three-term backpropagation algorithm

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

an Artificial Neural Network classifier is a nonparametric classifier. It does not need any priori knowledge regarding the statistical distribution of the class in a giver selected data Source. While, neural network can be trained to distinguish the criteria used to classify easily in a generalized manner that allows successful classification the newly arrived inputs not used during training. Through this paper it is eastliblished that back propagation neural network works successfully for the purpose of classification. Back propagation suffers from getting stuck into Local Minima. Weight optimization in Back propagation can be optimized using the Genetic Algorithm (GA). The back propagation algorithm is improved by invoking Genetic algorithm, to improve the overall performance of the classifier. The performance of a fitness algorithm using the approach suggested by us is a Hybrid System that is being analyzed in this paper. In this paper the issue of improving the fitness (weight adjustment) of Back propagation algorithm is addressed. Some of the Advantages of Hybrid algorithms are: convergence speed will be increased and the local minima problem can be overcome. The proposed Hybrid Algorithm is to perform learning as a back propagation and optimize weights using GA for classification.

Back Propagation (BP) algorithm is one of the oldest learning techniques used by artificial neural networks (ANN). It has successfully been implemented in various practical problems. However, the algorithm still faces some drawbacks such as getting easily stuck at local minima and needs longer time to converge on an acceptable solution. Recently, the introduction of Second Order Methods has shown a significant improvement on the learning in BP but it still has some drawbacks such as slow convergence and complexity. To overcome these limitations, this research proposed a modified approach for BP by introducing the Conjugate Gradient and Quasi-Newton which were Second Order methods together with ‘gain’ parameter. The performances of the proposed approach is evaluated in terms of lowest number of epochs, lowest CPU time and highest accuracy on five benchmark classification datasets such as Glass, Horse, 7Bit Parity, Indian Liver Patient and Lung Cancer. The results show that the proposed Second Order methods with ‘gain’ performed better than the BP algorithm.

BP(Back Propagation) neural network, as a method of data fusion technology, has been used in many common fields widely. While, the main problem of BP algorithm is that the optimal procedure is easily trapped into local minimum value and the speed of convergence is very slow. To avoid this problem, this paper, which, making use of ergodicity property of chaos, starts its improvement from the learning rate. Validity of the proposed method is examined by performing simulations on network traffic prediction, the result shows that the improved algorithm not only is more efficient in internet traffic prediction with higher precision and faster speed of convergence, but also somewhat saves the network from the problem of local minima.

In applying neural network to identification, back propagation (BP) algorithm is usually trapped to a local optimum and has a low speed of convergence, whereas particle swarm optimization (PSO) is advantageous in terms of global optimal searching. In this paper, a new algorithms which combines guaranteed convergence particle swarm optimizer (GCPSO) algorithm with BP (Back Propagation) algorithm is introduced in this paper and applied to BP neural networks to optimize the parameters of BP neural networks so as to improve the convergence speed and precision of BP neural networks. Compared with the BP algorithm and PSO-BP algorithm, the results of the simulation show that the algorithm of BP Neural Network Model Based on Guaranteed Convergence Particle Swarm Optimization speeds up the convergence process and enhances the accurate rate in pattern recognition.

Backpropagation is a supervised learning algorithm for training multi-layer neural networks for function approximation and pattern classification by minimizing a suitably defined error metric (e.g., the mean square error between the desired and actual outputs of the network for a training set) using gradient descent. It does this by calculating the partial derivative of the overall error and changing each weight by a small amount (determined by the learning rate) in a direction that is expected to reduce the error. Despite its success on a number of real-world problems, backpropagation can be very slow (it requires hundreds of passes (epochs) through the training set). Also, its performance is extremely sensitive to the choice of parameters such as the learning rate. The mathematical considerations that go into the derivation of backpropagation require that the learning rate be as small as possible. On the other hand, in order reduce the number training epochs required to learn the desired input-output mapping, it is desirable to make the weight changes as large as possible without causing the error to increase. Also it is desirable to have an algorithm that can change the learning rate dynamically so that it is close to optimal (in terms of reducing the error as much as possible given the local gradient information) at each epoch (thereby eliminating the reliance on guesswork in the choice of the learning rate). Several authors have proposed methods for adaptively adjusting the learning rate based on certain assumptions about the shape of the error surface (e.g., quadratic) and/or estimation of higher order derivatives of the error with respect to the weights at a given point in the weight space. The primary disadvantage of such methods is that the estimation of second order derivatives is not only computationally expensive but also prone to inaccuracy due to the approximation of derivatives by discrete differences. In this paper we propose and evaluate a heuristically motivated method for adaptive modification of the learning rate in backpropagation that does not require the estimation of higher order derivatives. We present a modified version of the Backpropagation learning algorithm which uses a simple heuristic to come up with a learning parameter value at each epoch. We present numerous simulations on real-world data sets, using our modified algorithm. We compare these results with results got through standard backpropagation learning algorithm, and also various modifications of the standard backpropagation algorithm (e.g., flat-spot elimination methods) that have been discussed in the literature. Our simulation results suggest that the adaptive learning rate modification helps substantially speed up the convergence of backpropagation algorithm. Furthermore, it makes the initial choice of the learning rate fairly unimportant as our method allows the learning rate to change and settle at a reasonable value for the specific problem. As a standard against which to compare our results, we computed the quasi-optimal value of the learning parameter at each epoch. Simulation results indicate that our heuristic modification matches the performance of backpropagation with the quasi-optimal learning rate. The computational complexity of this algorithm is analysed and compared with that of standard backpropagation.

The conventional back-propagation algorithm is basically a gradient-descent method, it has the problems of local minima and slow convergence. A new generalized back-propagation algorithm which can effectively speed up the convergence rate and reduce the chance of being trapped in local minima is introduced. The new back-propagation algorithm is to change the derivative of the activation function so as to magnify the backward propagated error signal, thus the convergence rate can be accelerated and the local minimum can be escaped. In this letter, we also investigate the convergence of the generalized back-propagation algorithm with constant learning rate. The weight sequences in generalized back-propagation algorithm can be approximated by a certain ordinary differential equation (ODE). When the learning rate tends to zero, the interpolated weight sequences of generalized back-propagation converge weakly to the solution of associated ODE.

There are a number of problems associated with training neural networks with backpropagation algorithm. The algorithm scales exponentially with increased complexity of the problem. It is very often trapped in local minima, and is not robust to changes of network parameters such as number of hidden layer neurons and learning rate. The use of genetic algorithms is a recent trend, which is good at exploring a large and complex search space, to overcome such problems. In this paper a genetic algorithm is proposed for training feedforward neural networks and its performances is investigated. The results are analyzed and compared with those obtained by the backpropagation algorithm.

Accelerating neural network training using weight extrapolations

Price forecasting has become essential tool in deregulated electricity market. It is used by utility operators for bidding in the competitive market to increase their profits and services. The models for electricity price forecasting can be mainly categorized into (i) Statistical models, (ii) Artificial Intelligence models & (iii) Hybrid models. AI based models, i.e., ANN have gained popularity in the recent years due to their black box nature and non-linear mapping capability, it is generally trained using back propagation algorithm, which has got some serious limitations like large training time, local minima etc. In this work, comparison of various training algorithms for ANN has been presented for accurate electricity price forecasting problem, the results obtained are contrasted with the generic ANN model trained using back propagation (BP) algorithm and other popular training algorithm. It is found that the generic ANN with Levenberg-Marquardt (LM) training method provides better result as compared to other models.

To address the insufficient expressive capabilities of traditional methods in assessing the development level of rural tourism, this study explores the fusion application of the Bidirectional Encoder Representations from Transformers (BERT) and the Back Propagation (BP) algorithm to enhance the accuracy and comprehensiveness of rural tourism development assessment. Firstly, this study introduces the BERT deep learning model and its applications in natural language processing, alongside the role of the BP algorithm in pattern recognition and predictive analysis. Subsequently, a framework for assessing rural tourism development levels, integrating BERT and the BP algorithm, is proposed. This framework collects multidimensional rural tourism-related data and utilizes the BERT model for sentiment analysis and topic extraction from textual data. Empirical analysis of rural tourism development in a specific region validates the effectiveness of the proposed approach. Experimental results demonstrate: (1) The model achieves an accuracy of 84.33% and an F1 score of 85.33% on the publicly available Laptop dataset, with a processing time of 20 s, significantly outperforming other methods. Compared to traditional approaches, the proposed method accurately captures correlations between textual information and numerical data, thereby enhancing the credibility and accuracy of assessment results. (2) From the ablation study results, it is evident that removing any component from the model leads to performance degradation. Specifically, removing the Bidirectional Gated Recurrent Unit (BiGRU) reduces accuracy and F1 scores to 78.21% and 76.33% on the Laptop dataset, and to 85.10% and 70.45% on the Tourist_F dataset. Removing Text Convolutional Neural Network (CNN) reduces accuracy and F1 scores to 79.34% and 77.56% on the Laptop dataset, and to 86.25% and 72.11% on the Tourist_F dataset. The most significant performance decline occurs upon removing BERT, with accuracy and F1 scores decreasing to 70.14% and 66.43% on the Laptop dataset, and to 78.45% and 60.33% on the Tourist_F dataset. These results indicate that BiGcRU, TextCNN, and BERT each contribute significantly to the model’s performance. This study provides substantial support for advancing sustainable development in rural tourism, offering significant practical and innovative value.

Parameter estimation of fuzzy neural network controller based on a modified differential evolution

Ant colony optimization (ACO) algorithm has the powerful ability of searching the global optimal solution, and backpropagation (BP) algorithm has the feature of rapid convergence on the local optima. The proper hybrid of the two algorithms (ACO-BP) may accelerate the evolving speed of neural networks and improve the forecasting precision of the well-trained networks. ACO-BP scheme adopts ACO to search the optimal combination of weights in the solution space, and then uses BP algorithm to obtain the accurate optimal solution quickly. The ACO-BP and BP algorithms were applied to the problems of function approaching and modeling quantitative structure-activity relationships of Herbicides. Experiment results show that the proposed ACO-BP scheme is more efficient and effective than BP algorithm. Furthermore, ACO-BP reliably performs well when the number of hidden nodes varies.KeywordsHide NodeEvolve Neural NetworkForecast PrecisionPheromone TablePheromone IntensityThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Performance Evaluation of Back propagation Algorithm by change in learning rate and momentum value – A Machine learning Approach

Rainfall is a very important factor in agriculture and development planning. This study aims to predict rainfall and rainfall properties in Medan using a multilayer neural network with Backpropagation and Learning Vector Quantization (LVQ) algorithms. Rainfall data used for training is rainfall data for the last 30 years. The parameters used in this study consisted of two inputs namely the month and the volume of rainfall. The training process produces the best architecture with 3 layer three for Backpropagation and two layers for Learning Vector Quantization with learning rate 0.5. From the prediction results obtained Backpropagation algorithm more accurate in predicting rainfall data last 30 years compared with LVQ with an average difference of 21.99%. Backpropagation and LVQ algorithms have better accuracy in the dry season, with accuracy for Backpropagation algorithm between 75 - 99% and LVQ algorithm of 60 - 82%. For both algorithms, the influence of El-Nino and La-Nina phenomena is not so significant.