Using adaptive learning rate to generate adversarial images

Like many optimization algorithms, stochastic variational inference is sensitive to the choice of the learning rate. If the learning rate is too small, the optimization process may be slow, and the algorithm might get stuck in local optima. On the other hand, if the learning rate is too large, the algorithm may oscillate or diverge, failing to converge to a solution. Adaptive learning rate methods such as Adam, AdaMax, Adagrad, or root mean square propagation automatically adjust the learning rate based on the history of gradients. Nevertheless, if the base learning rate is too large, the variational parameters might still oscillate around the optimal solution. With learning rate schedules, the learning rate can be reduced gradually to mitigate this problem. However, the amount at which the learning rate should be decreased in each iteration is not known a priori , which can significantly impact the performance of the optimization. In this work, we propose a method to decay the learning rate based on the history of the variational parameters. We use an empirical measure to quantify the amount of oscillations against the progress of the variational parameters to adapt the learning rate. The approach requires little memory and is computationally efficient. We demonstrate in various numerical examples that our method reduces the sensitivity of the optimization performance to the learning rate and that it can also be used in combination with other adaptive learning rate methods.

In general, backpropagation neural networks converge faster with adaptive learning rates than with learning rates that remain constant or grow or decay disregarding to the network error. This is because each synapse has its own learning rate that can vary over time by an amount appropriate to that weight. However, adaptive learning rates cause neural networks to saturate during training. When learning rates are permitted to assume values greater than unity, they are considered "greedy". Greedy adaptive learning rates can reduce not only the training times of networks, but also compromise the stability of the training process, leading to a network that fails to converge. This paper proposes a simple ad hoc approach called "shocking" as a partial solution to the instability problem caused by greedy adaptive learning rates. An analysis based on training times and failure rates for two inherently unstable benchmark problems is used to validate the use of shocking.

Adaptive learning rate of SpikeProp based on weight convergence analysis

Adaptive gradient methods such as RMSprop and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives. However, Adam can have undesirable convergence behaviors due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers [27]. In this paper, we propose an adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate. This results in a faster adaptation to the local gradient variance, which leads to more desirable empirical convergence behaviors than Adam. We prove the proposed algorithm converges under mild assumptions for nonconvex stochastic optimization problems, and demonstrate the improved efficacy of our adaptive averaging approach on machine translation, natural language understanding and large-batch pretraining of BERT. The code is available at https://github.com/zhuchen03/MaxVA.

Adaptive proximal SGD based on new estimating sequences for sparser ERM

As a key parameter in the sintering process, the ferrous oxide content of sinter can reflect the working condition, energy consumption level, and quality level of the final sintered products in the sintering process. It has become a key problem to realize the prediction of ferrous oxide content in sinter and feedback control of sinter quality accordingly. The two commonly used methods for detecting ferrous oxide content in industrial production currently do not meet real-time requirements and cannot provide timely feedback for production regulation. Therefore, research on real-time prediction technology of ferrous oxide content in sinter was carried out, and an optimized back propagation neural network model was established to realize the mapping between characteristic parameters and the FeO content in sinter. The characteristic parameters include image parameters and process parameters. Through the research on the brightness change trend of the machine tail cross-section image, the best cross-section image acquisition method based on brightness difference is realized, and image parameters are obtained by image processing technology. The process parameters were selected using correlation analysis. Through data processing techniques such as data cleaning, normalization, and feature fusion, feature parameters were obtained as input vectors for the neural network. To improve prediction accuracy and system stability, an adaptive learning rate and genetic algorithm were used to optimize the traditional BP neural network. The average test error of the optimized prediction model was 0.32%. Taking actual data production as an example, test data on the FeO content of sinter were extracted from the laboratory. Compared with the FeO content predicted by the system, the prediction time of the system was about 2 h earlier than the test time. In terms of prediction accuracy, the average absolute error was 0.25%, and the absolute prediction error was not more than ±1%.

This paper presents a tolerance analog circuit hard fault and soft fault diagnosis method based on the BP neural network and particle swarm optimization algorithm. First, select the mean square error function of BP neural network as the fitness function of the PSO algorithm. Second, change the guidance of neural network algorithms rely on gradient information to adjust the network weights and threshold methods, through the use of the characteristics of the particle swarm algorithm groups parallel search to find more appropriate network weights and threshold. Then using the adaptive learning rate and momentum BP algorithm to train the BP neural network. Finally, the network is applied to fault diagnosis of analog circuit, can quickly and effectively to the circuit fault diagnosis.

Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

In the learning process carried out by Backpropagation the learning function is important in finding optimal results. This study aims to optimize the learning function of artificial neural networks in increasing the accuracy of coffee export predictions according to the main destination countries as research objects. This study applies the learning function to weights in Matlab, namely Gradient Descent with Adaptive Learning Rate (traingda), Gradient Descent with Momentum (traingdm), and Gradient Descent with Momentum and Adaptive Learning Rate (trainingdx) using several hidden layers, namely 15,30 and 45. Based on a series of trials conducted, the results of the study show that by implementing the Gradient Descent learning function with an Adaptive Learning Rate (trainingda) with a hidden layer of 30 it is capable of training neural networks with a better level of optimization, performing 143 iterations which produces a truth accuracy of 83%. When compared with the use of other learning functions that only last with an accuracy of no more than 78%. In general, it can be concluded that the optimization of the Gradient Descent learning function with Adaptive Learning Rate (trainda) can be applied to predict coffee exports according to the main destination countries, because the iterative process carried out to achieve convergence in increasing accuracy performs well

Recent theories propose that autism is characterized by an impairment in determining when to learn and when not. Here, we investigated this hypothesis by estimating learning rates (i.e. the speed with which one learns) in three different environments that differed in rule stability and uncertainty. We found that neurotypical participants with more autistic traits performed worse in a volatile environment (with unstable rules), as they chose less often for the most rewarding option. Exploratory analyses indicated that performance was specifically worse when reward rules were opposite to those initially learned for participants with more autistic traits. However, there were no differences in the adjustment of learning rates between participants with more versus less autistic traits. Together, these results suggest that performance in volatile environments is lower in participants with more autistic traits, but that this performance difference cannot be unambiguously explained by an impairment in adjusting learning rates.

The back propagation (BP) algorithm is a very popular learning approach in feedforward multilayer perceptron networks. However, the most serious problem associated with the BP is local minima problem and slow convergence speeds. Over the years, many improvements and modifications of the back propagation learning algorithm have been reported. In this research, we propose a new modified back propagation learning algorithm by introducing adaptive gain together with adaptive momentum and adaptive learning rate into weight update process. By computer simulations, we demonstrate that the proposed algorithm can give a better convergence rate and can find a good solution in early time compare to the conventional back propagation. We use two common benchmark classification problems to illustrate the improvement in convergence time.Keywordsback propagationconvergence speedadaptive gainadaptive momentumadaptive learning rate

Effective neural network training with adaptive learning rate based on training loss

The Correlation filtering algorithm is not effective for fast deformation and fast movement. It is easy to lose when encountering problems such as occlusion. However, it has a good advantage of dealing with situations such as motion blur and lighting changes. A tracking algorithm based on color statistical features has a good effect on the rotation and translation of objects. The Staple algorithm combines the two algorithms to track using complementary fusion, but it also does not handle the occlusion and other issues well. In this paper, based on the Staple algorithm, the average peak correlation energy (APCE) and the maximum response are introduced. The value is used as the tracking confidence, and a detector using a support vector machine (SVM) is added. When the tracking confidence is low, the target is blocked or moved violently. At this time, the detector works, and the search area is expanded around the original area for the target. At the same time, because the traditional tracking algorithm uses a fixed learning rate to update the template, this paper uses an adaptive tracking learning rate. When the tracking confidence is low, the update speed of the target model is reduced, which can effectively deal with the occlusion deformation in the tracking process. OTB100 benchmark experiments show that this method can solve the occlusion problem during target tracking. The degree of change is robust and stability.

Classification of Bisyllabic Lexical Stress Patterns Using Deep Neural Networks

In this paper, we consider a joint design of beamforming vector and learning rate in MIMO over-the-air computation (AirComp) for federated learning. Since the learning performance improves with adaptive learning rates, we jointly optimize the receive beamforming vector and the learning rates. We first demonstrate the AirComp-multicasting duality between the uplink AirComp receive beamforming for federated learning systems and the downlink transmit beamforming for multicast systems. We design a low-complexity algorithm based on the projected subgradient method of the dual problem. Numerical results show that the proposed algorithm achieves nearly the same performance as the ideal federated learning system without aggregation errors.