Dynamic learning rate decay for stochastic variational inference

Convolutional neural networks (CNNs) have proved their efficiency in performing image classification tasks, as they can automatically extract the image features and make the corresponding prediction. Meanwhile, the CNNs application is highly challenged by their vulnerability to adversarial samples. These samples are slightly different from the legitimate samples, but the CNN gives wrong classification. There are various ways to find the adversarial samples. The most common method is using backpropagation to generate gradients as the directed perturbation. Contrarily to set a constrained limitation, in this paper, we use iterative fast gradient sign method to generate adversarial images with the minimum perturbation. The CNNs were trained to perform surgical tool recognition as a configuration for the modern operation room. The coefficient or the learning rate which influenced the modification per iteration, was set to be adaptive instead of a fixed number. A few functions were utilized to perform the learning rate decay to compare the performance. Especially, we propose a new adaptive learning rate algorithm that consider the loss as a part of influence factor constitute the learning rate for the rest iterations. According to the experiments, our loss adaptive learning rate method was proved to be efficient to get the minimal perturbations for adversarial attack.

In general, backpropagation neural networks converge faster with adaptive learning rates than with learning rates that remain constant or grow or decay disregarding to the network error. This is because each synapse has its own learning rate that can vary over time by an amount appropriate to that weight. However, adaptive learning rates cause neural networks to saturate during training. When learning rates are permitted to assume values greater than unity, they are considered "greedy". Greedy adaptive learning rates can reduce not only the training times of networks, but also compromise the stability of the training process, leading to a network that fails to converge. This paper proposes a simple ad hoc approach called "shocking" as a partial solution to the instability problem caused by greedy adaptive learning rates. An analysis based on training times and failure rates for two inherently unstable benchmark problems is used to validate the use of shocking.

Optimizers in Convolutional Neural Networks play an important role in many advanced deep learning models. Studies on advanced optimizers and modifications of existing optimizers continue to hold significant importance in the study of machine tools and algorithms. There are a number of studies to defend and the selection of these optimizers illustrate some of the challenges on the effectiveness of these optimizers. Comprehensive analysis on the optimizers and alteration with famous activation function Rectified Linear Unit (ReLU) offered to protect effectiveness. Significance is determined based on the adjustment with the original Softmax and ReLU. Experiments were performed with Adam, Root Mean Squared Propagation (RMSprop), Adaptive Learning Rate Method (Adadelta), Adaptive Gradient Algorithm (Adagrad) and Stochastic Gradient Descent (SGD) to examine the performance of Convolutional Neural Networks for image classification using the Canadian Institute for Advanced Research dataset (CIFAR-10).

The ensemble learning method is a necessary process that provides robustness and is more accurate than the single model. The snapshot ensemble convolutional neural network (CNN) has been successful and widely used in many domains, such as image classification, fault diagnosis, and plant image classification. The advantage of the snapshot ensemble CNN is that it combines the cyclic learning rate schedule in the algorithm to snap the best model in each cycle. In this research, we proposed the dropCyclic learning rate schedule, which is a step decay to decrease the learning rate value in every learning epoch. The dropCyclic can reduce the learning rate and find the new local minimum in the subsequent cycle. We evaluated the snapshot ensemble CNN method based on three learning rate schedules: cyclic cosine annealing, max-min cyclic cosine learning rate scheduler, and dropCyclic then using three backbone CNN architectures: MobileNetV2, VGG16, and VGG19. The snapshot ensemble CNN methods were tested on three aerial image datasets: UCM, AID, and EcoCropsAID. The proposed dropCyclic learning rate schedule outperformed the other learning rate schedules on the UCM dataset and obtained high accuracy on the AID and EcoCropsAID datasets. We also compared the proposed dropCyclic learning rate schedule with other existing methods. The results show that the dropCyclic method achieved higher classification accuracy compared with other existing methods.

Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

Adaptive learning rate of SpikeProp based on weight convergence analysis

Recent theories propose that autism is characterized by an impairment in determining when to learn and when not. Here, we investigated this hypothesis by estimating learning rates (i.e. the speed with which one learns) in three different environments that differed in rule stability and uncertainty. We found that neurotypical participants with more autistic traits performed worse in a volatile environment (with unstable rules), as they chose less often for the most rewarding option. Exploratory analyses indicated that performance was specifically worse when reward rules were opposite to those initially learned for participants with more autistic traits. However, there were no differences in the adjustment of learning rates between participants with more versus less autistic traits. Together, these results suggest that performance in volatile environments is lower in participants with more autistic traits, but that this performance difference cannot be unambiguously explained by an impairment in adjusting learning rates.

Adaptive gradient methods such as RMSprop and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives. However, Adam can have undesirable convergence behaviors due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers [27]. In this paper, we propose an adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate. This results in a faster adaptation to the local gradient variance, which leads to more desirable empirical convergence behaviors than Adam. We prove the proposed algorithm converges under mild assumptions for nonconvex stochastic optimization problems, and demonstrate the improved efficacy of our adaptive averaging approach on machine translation, natural language understanding and large-batch pretraining of BERT. The code is available at https://github.com/zhuchen03/MaxVA.

Remote Sensing (RS) image classification has recently attracted great attention for its application in different tasks, including environmental monitoring, battlefield surveillance, and geospatial object detection. The best practices for these tasks often involve transfer learning from pre-trained Convolutional Neural Networks (CNNs). A common approach in the literature is employing CNNs for feature extraction, and subsequently train classifiers exploiting such features. In this paper, we propose the adoption of transfer learning by fine-tuning pre-trained CNNs for end-to-end aerial image classification. Our approach performs feature extraction from the fine-tuned neural networks and remote sensing image classification with a Support Vector Machine (SVM) model with linear and Radial Basis Function (RBF) kernels. To tune the learning rate hyperparameter, we employ a linear decay learning rate scheduler as well as cyclical learning rates. Moreover, in order to mitigate the overfitting problem of pre-trained models, we apply label smoothing regularization. For the fine-tuning and feature extraction process, we adopt the Inception-v3 and Xception inception-based CNNs, as well the residual-based networks ResNet50 and DenseNet121. We present extensive experiments on two real-world remote sensing image datasets: AID and NWPU-RESISC45. The results show that the proposed method exhibits classification accuracy of up to 98%, outperforming other state-of-the-art methods.

Convolutional neural network (CNN) has gained increasing attention in fault classification. However, the performance of CNN is sensitive to its learning rate. Some previous works have been done to tune the learning rate, including the "trial and error" and manual search, which heavily depend on the experts' experiences and should be conducted repeatedly on every dataset. Because of the variety of the fault data, it is time-consuming and labor intensive to use these traditional tuning methods for fault classification. To overcome this problem, in this article, we develop a novel learning rate scheduler based on the reinforcement learning (RL) for convolutional neural network (RL-CNN) in fault classification, which can schedule the learning rate efficiently and automatically. First, a new RL agent is designed to learn the policies about the learning rate adjustment during the training process. Second, a new structure of RL-CNN is developed to balance the exploration and exploitation of the agent. Third, the bagging ensemble version of RL-CNN (RL-CNN-Ens) is presented. Three bearing datasets are used to test the performance of RL-CNN-Ens. The results show that RL-CNN-Ens outperforms the traditional DLs and machine learning methods. Meanwhile, RL-CNN-Ens can find the state-of-the-art learning rate schedulers as human designed, showing its potential in fault classification.

In the biaxially stretched film (BOPP) thickness control system, the traditional PID and Active-Disturbance Rejection Controller (ADRC) can’t achieve the ideal control effect. The Smith prediction method is used in the essay to establish a discretization model for the BOPP thickness control system. Combining with BP self-learning algorithm, a fast self-learning improved ADRC control algorithm (FSADRC) is proposed. By means of the additional momentum term and the adaptive learning rate method, the nonlinear combination of the ADRC system is adjusted in real time, the optimal control parameters are found, and the parameters are self-tuned. As a consequence, the improved algorithm is applied to the biaxial tensile film thickness control model. The simulation results show that the method has the advantages of high response speed and strong self-adaptive ability, which can effectively improve the control performance of the BOPP thickness control system.

Real-time adaptive impedance compensator using simultaneous perturbation stochastic approximation for enhanced physical human–robot interaction transparency

Electricity prediction helps electric power companies to generate sufficient electrical power to consumers. The primary source used in performing forecasting is historical electricity usage. This research identified the optimum historical load data period in generating the best model for short-term forecasting of a household. The experiment applied Long Short-Term Memory (LSTM) architecture using Adaptive Learning Rate Method (Adadelta) on four categories of dataset: one-year, two-years, three-years, and four-years. The models produced were evaluated using mean squared error (MSE) and mean absolute error (MAE). The model generated from two-years of historical data performed the best among all other models with MSE value of 0.133621 and MAE value of 0.050653. The experiment was enclosed with the application of the model to predict the electricity usage of the following year, shown in two sample categories: one day and one week. Then, the prediction results were compared with the actual load.KeywordsElectricityPredictionMachine learningLSTM

Deep Belief Networks (DBNs) have emerged as powerful tools for feature learning, representation, and generative modeling. This paper presents a comprehensive exploration of the various training algorithms employed in the training of DBNs. DBNs, composed of multiple layers of stochastic hidden units, have found applications in diverse domains such as computer vision, natural language processing, and bioinformatics. The paper begins by delving into the pre-training phase, where Restricted Boltzmann Machines (RBMs) play a central role. We review the Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) algorithms, shedding light on their strengths and weaknesses in initializing deep networks. Emphasis is placed on their applicability to different data types and scales. Moving to the fine-tuning stage, the paper explores the use of backpropagation with gradient descent, discussing modern optimization techniques, including stochastic gradient descent and adaptive learning rate methods. We also examine regularization techniques like dropout and weight decay to address overfitting concerns. Furthermore, we discuss architectural variants of DBNs, such as Convolutional Deep Belief Networks (CDBNs) for image data and Recurrent DBNs for sequential data. We highlight the adaptation of DBNs for specific tasks, including classification, regression, clustering, and generative modeling.

With the trend toward taller and more flexible building structures, a mass-damper shaking table system has been considered as means for vibration suppression to external excitation and disturbances in recent years. However, there are few researches on the control of nonlinear structure using active mass damper (AMD) under earthquake excitation, especially for high-rise building. In this work, a multilayer feedforward neural network with the modified Newton method, similar to BFGS algorithm, is developed for vibration suppression of a building structure. The benchmark tests show that the modified Newton method is superior to many conventional ones: steepest descent, steepest descent with adaptive learning rate, conjugate gradient, and Newton-based methods and is suitable to small network in engineering applications. Experimental results show that an AMD system combined with the modified Newton method remains effective for building structure vibration suppression under free vibration, forced vibration and Ji-Ji Earthquake (Sep. 21, 1999) excitation.