An Optimized Back Propagation Learning Algorithm with Adaptive Learning Rate

In this work, we investigate a communication-efficient multi-hierarchical signSGD (MH-signSGD) algorithm with an adaptive learning rate. Under the symmetric assumption of the stochastic gradient distribution, we show that, without the need for learning rate tuning, our proposed MH-signSGD matches the state-of-art sublinear convergence rate O(1/ $\sqrt K $) in nonconvex settings, where K is the number of iterations. Our adaptive learning strategy is based on stochastically approximating the learning rate found by greedily minimizing an error upper bound between two successive iterations. Moreover, by leveraging a normal approximation technique to characterize stochastic gradient sign error, we are able to sharpen the convergence analysis of MH-sighSGD with a fixed learning rate 1/ $\sqrt K $ and establish a strong result in the large-system regime, which says that the MH-signSGD algorithm asymptotically converges to a stationary point at rate O(1/$\sqrt M $), where M is the number of workers. In comparison, most existing work on signSGD can only prove a weaker finite neighborhood convergence in the large system regime. We validate our theoretical results experimentally both on synthetic data and real-world datasets.

The back propagation (BP) algorithm is a very popular learning approach in feedforward multilayer perceptron networks. However, the most serious problem associated with the BP is local minima problem and slow convergence speeds. Over the years, many improvements and modifications of the back propagation learning algorithm have been reported. In this research, we propose a new modified back propagation learning algorithm by introducing adaptive gain together with adaptive momentum and adaptive learning rate into weight update process. By computer simulations, we demonstrate that the proposed algorithm can give a better convergence rate and can find a good solution in early time compare to the conventional back propagation. We use two common benchmark classification problems to illustrate the improvement in convergence time.Keywordsback propagationconvergence speedadaptive gainadaptive momentumadaptive learning rate

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

Adaptive proximal SGD based on new estimating sequences for sparser ERM

In the learning process carried out by Backpropagation the learning function is important in finding optimal results. This study aims to optimize the learning function of artificial neural networks in increasing the accuracy of coffee export predictions according to the main destination countries as research objects. This study applies the learning function to weights in Matlab, namely Gradient Descent with Adaptive Learning Rate (traingda), Gradient Descent with Momentum (traingdm), and Gradient Descent with Momentum and Adaptive Learning Rate (trainingdx) using several hidden layers, namely 15,30 and 45. Based on a series of trials conducted, the results of the study show that by implementing the Gradient Descent learning function with an Adaptive Learning Rate (trainingda) with a hidden layer of 30 it is capable of training neural networks with a better level of optimization, performing 143 iterations which produces a truth accuracy of 83%. When compared with the use of other learning functions that only last with an accuracy of no more than 78%. In general, it can be concluded that the optimization of the Gradient Descent learning function with Adaptive Learning Rate (trainda) can be applied to predict coffee exports according to the main destination countries, because the iterative process carried out to achieve convergence in increasing accuracy performs well

Like many optimization algorithms, stochastic variational inference is sensitive to the choice of the learning rate. If the learning rate is too small, the optimization process may be slow, and the algorithm might get stuck in local optima. On the other hand, if the learning rate is too large, the algorithm may oscillate or diverge, failing to converge to a solution. Adaptive learning rate methods such as Adam, AdaMax, Adagrad, or root mean square propagation automatically adjust the learning rate based on the history of gradients. Nevertheless, if the base learning rate is too large, the variational parameters might still oscillate around the optimal solution. With learning rate schedules, the learning rate can be reduced gradually to mitigate this problem. However, the amount at which the learning rate should be decreased in each iteration is not known a priori , which can significantly impact the performance of the optimization. In this work, we propose a method to decay the learning rate based on the history of the variational parameters. We use an empirical measure to quantify the amount of oscillations against the progress of the variational parameters to adapt the learning rate. The approach requires little memory and is computationally efficient. We demonstrate in various numerical examples that our method reduces the sensitivity of the optimization performance to the learning rate and that it can also be used in combination with other adaptive learning rate methods.

In general, backpropagation neural networks converge faster with adaptive learning rates than with learning rates that remain constant or grow or decay disregarding to the network error. This is because each synapse has its own learning rate that can vary over time by an amount appropriate to that weight. However, adaptive learning rates cause neural networks to saturate during training. When learning rates are permitted to assume values greater than unity, they are considered "greedy". Greedy adaptive learning rates can reduce not only the training times of networks, but also compromise the stability of the training process, leading to a network that fails to converge. This paper proposes a simple ad hoc approach called "shocking" as a partial solution to the instability problem caused by greedy adaptive learning rates. An analysis based on training times and failure rates for two inherently unstable benchmark problems is used to validate the use of shocking.

Neural Networks have been used in system control, medicine, pattern recognition and business. The backpropagation neural network (BPNN) appear to be most popular and have been widely used in many applications. BPNN is a supervised learning technique for training multilayer feedforward neural networks. The gradient or steepest descent method is used to train a BPNN by adjusting the weights. The purpose of update numerical weights is minimize error of network between target and output. In this paper, focus with BPNN modeling with data battery for training and testing. We used discharge and Urban Dynamometer Driving Schedule (UDDS) as training data and testing data, respectively Architecture of BPNN consist of input layer, hidden layer and output layer. The otherhand, using BPNN has problem to define amount of hidden neurons. In this study, we used current or voltage as input in input layer, one hidden layer with 8 neurons and one output layer. We used Levenberg-Marquardt algorithm to get fast iteration when...

Deterministic cell-growth models describe the growth of cell populations using fixed mathematical rules, assuming no randomness in the system. These models are often based on differential equations that account for the rates of cell division, death, and other biological processes. The solution to the system is obtained via numerical methods. Most of the developed approaches are based on fixed step sizes. However, fixed step size implementation failed to offer the optimal solutions when dealing with stiff challenges. Fixed step size methods can be unstable for stiff equations, where some components of the solution change much more rapidly than others. The step size, required to maintain stability can become impractically small. Thus, the adaptive step size method is required. Adaptive step size methods adjust the step size dynamically based on the behavior of the solution, aiming to maintain a desired level of accuracy while optimizing computational efficiency. These methods are particularly useful for solving ordinary differential equations (ODEs) where the solution can vary rapidly in some regions and slowly in others. This study is devoted to comparing the implementation of fixed step size and adaptive step size in solving ordinary differential equations (ODEs). The fixed step size and adaptive step size numerical method are solved in this study via the fourth order Runge-Kutta method (RK4) and Runge-Kutta Fehlberg 45 method (RKF45). The performance of both numerical methods used will be analyzed by comparing the numerical results approximated with the actual data. Subsequently, the absolute error, relative error, and rounding-off error will be calculated to compare both approaches. Based on the more precise findings, this work has shown that adaptive step size is predicted to be the optimal representation for solving ODEs. As a result, this may help mathematicians to choose the most effective numerical approach for solving ODEs.

Error Back Propagation algorithm is one of the most popular method for training artificial neural networks. Unfortunately, this is also one of the slowest due to constant and small learning rate parameter used to update weights of neurons. There are therefore many methods for dynamic change this parameter in the adaptive manner. In this paper has been presented new method based on estimation of learning rate with respect of approximation for the value of output error.

A new approach is developed for synthetic aperture radar (SAR) automatic target recognition based on convolutional neural network (CNN). In order to improve the low efficiency caused by fixed learning rate in CNN, the AdaDelta method is introduced to adjust the learning rate adaptively. The experimental results on Moving and Stationary Target Acquisition and Recognition (MSTAR) data sets show that convergence is significantly faster in proposed method than CNN with fixed learning rate.

Convolutional neural networks (CNNs) have proved their efficiency in performing image classification tasks, as they can automatically extract the image features and make the corresponding prediction. Meanwhile, the CNNs application is highly challenged by their vulnerability to adversarial samples. These samples are slightly different from the legitimate samples, but the CNN gives wrong classification. There are various ways to find the adversarial samples. The most common method is using backpropagation to generate gradients as the directed perturbation. Contrarily to set a constrained limitation, in this paper, we use iterative fast gradient sign method to generate adversarial images with the minimum perturbation. The CNNs were trained to perform surgical tool recognition as a configuration for the modern operation room. The coefficient or the learning rate which influenced the modification per iteration, was set to be adaptive instead of a fixed number. A few functions were utilized to perform the learning rate decay to compare the performance. Especially, we propose a new adaptive learning rate algorithm that consider the loss as a part of influence factor constitute the learning rate for the rest iterations. According to the experiments, our loss adaptive learning rate method was proved to be efficient to get the minimal perturbations for adversarial attack.

A multilayer feedforward neural network with a modification to the standard backpropagation training of neural nets is considered. Information on the gradients of target outputs with respect to network inputs is used in this modification. This modified neural network is then applied for approximating design requirements of aerospace composite components, such as an aircraft engine guide vane and a satellite reflector assembly. This neural network functional approximator is then used with an optimization algorithm for determining the optimal parameters for component design requirements. The modified neural network provides for faster convergence of learning process and requires a smaller number of hidden layer nodes for convergence to similar training error levels. HE application of neural networks as universal approximators has been the focus of tremendous amount of activity in the past few years.16 Neural networks have been used for approximating and modeling both memoryless systems and dynamical systems with memory. This paper does not go into the various neural net architectures and training schemes, and the interested reader could refer to standard texts on this subject matter.1'2 This paper considers static neural networks (systems without memory) formed by interconnections without loops, called feedforward networks.2 In addition, this work considers a backpropagation learning scheme with adaptation of learning rate. There have been many additions to, or adaptations of, the basic backpropagation scheme, such as adaptive learning rate, learning with momentum, and the like,1'2 for improving convergence of the learning scheme, and these are not addressed in this paper. This paper explores a modification of the backpropagation learning scheme using information on the gradient of the target outputs with respect to the inputs of the network. The performance of the modified learning scheme is compared with that of the standard backpropagation learning scheme.2 A feedforward network with a single hidden layer trained with this modified learning scheme is applied in structural optimization problems. Figure 1 shows a feedforward net with a single hidden layer. The restriction to a single hidden layer is not a limitation of the modified learning scheme, and the new training scheme can be used on networks with multiple hidden layers also. The use of neural networks for structural response approximation has been reported by several authors.46 The primary reason for investigating such properties is to assess the applicability of trained neural networks for quick and accurate approximation of response functions during structural design optimization. Structural optimization, utilizing either gradient-based numerical search or genetic search algorithms, requires repeated function (response) evaluations during the search process. A single finite element analysis of large-scale structures requires a significant amount of computational time and resource and hence the need for robust approxima

Backpropagation is a supervised learning algorithm for training multi-layer neural networks for function approximation and pattern classification by minimizing a suitably defined error metric (e.g., the mean square error between the desired and actual outputs of the network for a training set) using gradient descent. It does this by calculating the partial derivative of the overall error and changing each weight by a small amount (determined by the learning rate) in a direction that is expected to reduce the error. Despite its success on a number of real-world problems, backpropagation can be very slow (it requires hundreds of passes (epochs) through the training set). Also, its performance is extremely sensitive to the choice of parameters such as the learning rate. The mathematical considerations that go into the derivation of backpropagation require that the learning rate be as small as possible. On the other hand, in order reduce the number training epochs required to learn the desired input-output mapping, it is desirable to make the weight changes as large as possible without causing the error to increase. Also it is desirable to have an algorithm that can change the learning rate dynamically so that it is close to optimal (in terms of reducing the error as much as possible given the local gradient information) at each epoch (thereby eliminating the reliance on guesswork in the choice of the learning rate). Several authors have proposed methods for adaptively adjusting the learning rate based on certain assumptions about the shape of the error surface (e.g., quadratic) and/or estimation of higher order derivatives of the error with respect to the weights at a given point in the weight space. The primary disadvantage of such methods is that the estimation of second order derivatives is not only computationally expensive but also prone to inaccuracy due to the approximation of derivatives by discrete differences. In this paper we propose and evaluate a heuristically motivated method for adaptive modification of the learning rate in backpropagation that does not require the estimation of higher order derivatives. We present a modified version of the Backpropagation learning algorithm which uses a simple heuristic to come up with a learning parameter value at each epoch. We present numerous simulations on real-world data sets, using our modified algorithm. We compare these results with results got through standard backpropagation learning algorithm, and also various modifications of the standard backpropagation algorithm (e.g., flat-spot elimination methods) that have been discussed in the literature. Our simulation results suggest that the adaptive learning rate modification helps substantially speed up the convergence of backpropagation algorithm. Furthermore, it makes the initial choice of the learning rate fairly unimportant as our method allows the learning rate to change and settle at a reasonable value for the specific problem. As a standard against which to compare our results, we computed the quasi-optimal value of the learning parameter at each epoch. Simulation results indicate that our heuristic modification matches the performance of backpropagation with the quasi-optimal learning rate. The computational complexity of this algorithm is analysed and compared with that of standard backpropagation.

The gradient descent backpropagation learning algorithm is based on minimizing the mean square error. An alternate approach to gradient descent is the exponentiated gradient descent algorithm which minimizes the relative entropy. Exponentiated gradient descent applied to backpropagation is proposed for a multilayer feedforward neural network. The learning rules for changing weights in the output layer as well the hidden layer neurons in the network are developed. Simulations were performed to explore the convergence and learning of the backpropagation algorithm with exponentiated gradient descent. Accuracy obtained with exponentiated gradient descent back propagation was comparable to the gradient descent back propagation while convergence was faster. The results show that exponentiated gradient descent can be extended to a multilayer feedforward neural network and used in pattern classification applications.