Response surface methodology's steepest ascent and step size revisited

This paper presents an improved artificial neural network (ANN) training using response surface methodology (RSM) optimization for membrane flux prediction. The improved ANN utilizes the design of experiment (DoE) technique to determine the neural network parameters. The technique has the advantage of training performance, with a reduced training time and number of repetitions in achieving good model prediction for the permeate flux of palm oil mill effluent. The conventional training process is performed by the trial-and-error method, which is time consuming. In this work, Levenberg–Marquardt (lm) and gradient descent with momentum (gdm) training functions are used, the feed-forward neural network (FFNN) structure is applied to predict the permeate flux, and airflow and transmembrane pressure are the input variables. The network parameters include the number of neurons, the learning rate, the momentum, the epoch, and the training functions. To realize the effectiveness of the DoE strategy, central composite design is incorporated into neural network methodology to achieve both good model accuracy and improved training performance. The simulation results show an improvement of more than 50% of training performance, with less repetition of the training process for the RSM-based FFNN (FFNN-RSM) compared with the conventional-based FFNN (FFNN-lm and FFNN-gdm). In addition, a good accuracy of the models is achieved, with a smaller generalization error.

Speech recognition is to convert speech signal into text. It is challenging task due to natural variations present in human speech and also due to background noise. Now a day’s researchers focus on deep learning due to it’s effectiveness and high performance. There is need to work on recognition of speech for mathematical words; many of the currently available methods proposed by researchers are not yet adequate to recognize speech for mathematical words. In this paper we focus on CNN (Convolution Neural Network) for recognition of connected word speech recognition for mathematical words plus, minus, square and square-root. The dataset of spoken words are created using Audacity. CNN model is verified by considering Adam, Gradient descent and Adagrad optimizer with learning rate of 0.001 and 0.0001. Result shows that Adam and Gradient descent optimizer gives better result for 0.001 learning rate.

Backpropagation is a supervised learning algorithm for training multi-layer neural networks for function approximation and pattern classification by minimizing a suitably defined error metric (e.g., the mean square error between the desired and actual outputs of the network for a training set) using gradient descent. It does this by calculating the partial derivative of the overall error and changing each weight by a small amount (determined by the learning rate) in a direction that is expected to reduce the error. Despite its success on a number of real-world problems, backpropagation can be very slow (it requires hundreds of passes (epochs) through the training set). Also, its performance is extremely sensitive to the choice of parameters such as the learning rate. The mathematical considerations that go into the derivation of backpropagation require that the learning rate be as small as possible. On the other hand, in order reduce the number training epochs required to learn the desired input-output mapping, it is desirable to make the weight changes as large as possible without causing the error to increase. Also it is desirable to have an algorithm that can change the learning rate dynamically so that it is close to optimal (in terms of reducing the error as much as possible given the local gradient information) at each epoch (thereby eliminating the reliance on guesswork in the choice of the learning rate). Several authors have proposed methods for adaptively adjusting the learning rate based on certain assumptions about the shape of the error surface (e.g., quadratic) and/or estimation of higher order derivatives of the error with respect to the weights at a given point in the weight space. The primary disadvantage of such methods is that the estimation of second order derivatives is not only computationally expensive but also prone to inaccuracy due to the approximation of derivatives by discrete differences. In this paper we propose and evaluate a heuristically motivated method for adaptive modification of the learning rate in backpropagation that does not require the estimation of higher order derivatives. We present a modified version of the Backpropagation learning algorithm which uses a simple heuristic to come up with a learning parameter value at each epoch. We present numerous simulations on real-world data sets, using our modified algorithm. We compare these results with results got through standard backpropagation learning algorithm, and also various modifications of the standard backpropagation algorithm (e.g., flat-spot elimination methods) that have been discussed in the literature. Our simulation results suggest that the adaptive learning rate modification helps substantially speed up the convergence of backpropagation algorithm. Furthermore, it makes the initial choice of the learning rate fairly unimportant as our method allows the learning rate to change and settle at a reasonable value for the specific problem. As a standard against which to compare our results, we computed the quasi-optimal value of the learning parameter at each epoch. Simulation results indicate that our heuristic modification matches the performance of backpropagation with the quasi-optimal learning rate. The computational complexity of this algorithm is analysed and compared with that of standard backpropagation.

A statistical optimization of methylene blue removal from aqueous solutions by Agaricus Campestris using multi-step experimental design with response surface methodology: Isotherm, kinetic and thermodynamic studies

Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

In recent years, control research has strongly highlighted the issue of training stability in the identification of non‐linear systems. This paper investigates the stability analysis of an interval type‐2 adaptive neuro‐fuzzy inference system (IT2ANFIS) as an identifier through a novel Lyapunov function. In so doing, stability analysis is initially conducted on the IT2ANFIS identifier, while performing the online training of both the antecedent and the consequent parameters by the gradient descent (GD) algorithm. In addition, the same stability analysis is carried out when the antecedent and the consequent parameters are trained by GD and forgetting factor recursive least square (FRLS) algorithms, respectively (GD + FRLS). A novel Lyapunov function is proposed in this study in order for the identifier stability to attain the required conditions. These conditions determine the permissible boundaries for the covariance matrix and the learning rates at every iteration of the identification procedure. Stability analysis reveals that wide range of learning rates is obtained. Furthermore, simulation results indicate that when the permissible boundaries are selected according to the proposed stability analysis, a stable identification process with appropriate performance is achieved.

To minimize the continuous and differentiable objective function f, the gradient descent, a first-order iterative method, can provide a sequence number {x_i} that converges to the minimum. The iteration can be written as x_(i+1) = x_i - t ∇f_i where t is a learning rate and i is an iterative number. However, the value of the learning rate affects the whole algorithm. If the value is too small, the sequence {x_i} will slowly converge, and if the value is too large, the sequence {x_i} will oscillate or diverge. To prevent this problem, the learning rate t is adjusted for each iteration i. Therefore, the problem is reduced to minimize f ̃(t_i ) = f(x_i - t_i ∇f_i). The line search method will choose the optimal t_i be the learning rate in iteration i. In this project, we improve the backtracking line search by choosing a smaller learning rate for gradient descent based on the previous iteration. After the experiment, the modified method performs as well as the standard method for simple functions and better in more complex functions in terms of running time and number of iterations.

In recent years, artificial neural networks have developed into a powerful tool for dealing with a multitude of problems for which classical solution approaches reach their limits. However, it is still unclear why randomly initialized gradient descent optimization algorithms, such as the well-known batch gradient descent, are able to achieve zero training loss in many situations even though the objective function is non-convex and non-smooth. One of the most promising approaches to solving this problem in the field of supervised learning is the analysis of gradient descent optimization in the so-called overparameterized regime. In this article we provide a further contribution to this area of research by considering overparameterized fully connected shallow artificial neural networks with piecewise affine activation, such as the rectified linear unit activation. Specifically, given that the activation function is not affine and the training input data are pairwise distinct, we show that, with high probability, the mean squared error using batch gradient descent optimization applied to such a randomly initialized artificial neural network converges to zero at a linear convergence rate as long as the width of the artificial neural network is large enough and the learning rate is small enough.

Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb() are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

The limit analysis and response surfaces method were combined to investigate the reliability of pressurized tunnel faces subjected to seismic force. The quasi-static method was utilized to introduce seismic force into the tunnel face. A 3D horn failure mechanism of pressurized tunnel faces subjected to seismic force was constructed. The collapse pressure of pressurized tunnel faces was solved by the kinematical approach. The limit state equation of pressurized tunnel faces was obtained according to the collapse pressure and support pressure. And then a reliability model of pressurized tunnel faces was established. The feasibility and superiority of the response surfaces method was verified by comparing with the Monte Carlo method. The influence of the mean of soil parameters and support pressure, variation coefficients, distribution type and correlation of c-phi on the reliability of pressurized tunnel faces was discussed. The reasonable safety factor and support pressure required by pressurized tunnel faces to satisfy 3 safety levels were presented. In addition, the effects of horizontal seismic force, vertical seismic force and correlation of kh-kv on the reliability of pressurized tunnel faces were also performed. The method of this work can give a new idea for anti-seismic design of pressurized tunnel faces.

We focus on the classification problem with a separable dataset, one of the most important and classical problems from machine learning. The standard approach to this task is logistic regression with gradient descent (LR+GD). Recent studies have observed that LR+GD can find a solution with arbitrarily large step sizes, defying conventional optimization theory. Our work investigates this phenomenon and makes three interconnected key observations about LR+GD with large step sizes. First, we find a remarkably simple explanation of why LR+GD with large step sizes solves the classification problem: LR+GD reduces to a batch version of the celebrated perceptron algorithm when the step size tends to infinity. Second, we observe that larger step sizes lead LR+GD to higher logistic losses when it tends to the perceptron algorithm, but larger step sizes also lead to faster convergence to a solution for the classification problem, meaning that logistic loss is an unreliable metric of the proximity to a solution. Surprisingly, high loss values can actually indicate faster convergence. Third, since the convergence rate in terms of loss function values of LR+GD is unreliable, we examine the iteration complexity required by LR+GD with large step sizes to solve the classification problem and prove that this complexity is suboptimal. To address this, we propose a new method, Normalized LR+GD – based on the connection between LR+GD and the perceptron algorithm – with much better theoretical guarantees.

Image registration is often used in the clinic, for example during radiotherapy and image-guide surgery, but also for general image analysis. Currently, this process is often very slow, yet for intra-operative procedures the speed is crucial. For intensity-based image registration, a nonlinear optimization problem should be solved, usually by (stochastic) gradient descent. This procedure relies on a proper setting of a parameter which controls the optimization step size. This parameter is difficult to choose manually however, since it depends on the input data, optimization metric and transformation model. Previously, the Adaptive Stochastic Gradient Descent (ASGD) method has been proposed that automatically chooses the step size, but it comes at high computational cost. In this paper, we propose a new computationally efficient method to automatically determine the step size, by considering the observed distribution of the voxel displacements between iterations. A relation between the step size and the expectation and variance of the observed distribution is then derived. Experiments have been performed on 3D lung CT data (19 patients) using a nonrigid B-spline transformation model. For all tested dissimilarity metrics (mean squared distance, normalized correlation, mutual information, normalized mutual information), we obtained similar accuracy as ASGD. Compared to ASGD whose estimation time is progressively increasing with the number of parameters, the estimation time of the proposed method is substantially reduced to an almost constant time, from 40 seconds to no more than 1 second when the number of parameters is 10⁵.

The effect of digital implementation on the gradient (steepest descent) algorithm commonly used in the mean-square adaptive equalization of pulse-amplitude modulated data signals is considered. It is shown that digitally implemented adaptive gradient algorithms can exhibit effects which are significantly different from those encountered in analog (infinite precision) algorithms. This is illustrated by considering the often quoted result of stochastic approximation that to achieve the optimum rate of convergence in an adaptive algorithm the step size should be proportional to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/n</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> is the number of iterations. On closer examination one finds that this result applies only when <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> is large and is relevant only for analog algorithms. It is shown that as the number of iterations becomes large one should not continually decrease the step size in a digital gradient algorithm. This result is a manifestation of the quantization inherent in any digitally implemented system. A surprising result is that these effects produce a digital residual mean-square error that is minimized by making the step size as large as possible. Since the analog residual error is minimized by taking small step sizes, the optimum step-size sequence reflects a compromise between these competing goals. The performance of a time-varying gain sequence suggested by stochastic approximation is contrasted with the performance of a constant step-size sequence. It is shown that in a digital environment the latter sequence is capable of attaining a smaller residual error.

Much recent attention has been devoted to gradient descent algorithms where the steepest descent step size is replaced by a similar one from a previous iteration or gets updated only once every second step, thus forming a faster gradient descent method. For unconstrained convex quadratic optimization these methods can converge much faster than steepest descent. But the context of interest here is application to certain ill-posed inverse problems, where the steepest descent method is known to have a smoothing, regularizing effect, and where a strict optimization solution is not necessary. Specifically, in this paper we examine the effect of replacing steepest descent by a faster gradient descent algorithm in the practical context of image deblurring and denoising tasks. We also propose several highly efficient schemes for carrying out these tasks independently of the step size selection, as well as a scheme for the case where both blur and significant noise are present. In the above context there are situations where many steepest descent steps are required, thus building slowness into the solution procedure. Our general conclusion regarding gradient descent methods is that in such cases the faster gradient descent methods offer substantial advantages. In other situations where no such slowness buildup arises the steepest descent method can still be very effective.

The proposed approach combines program evaluation review technique (PERT) and critical path method (CPM) into a single-project scheduling problem. Given that activity time and cost are uncertain, a robust design for project scheduling using the response surface methodology (RSM) is presented in the proposed research. The controllable variables are the mean and tolerance of activity time, which are both cost related. In order to simulate an actual application, activity times, activity tolerances, activity time costs and activity tolerance costs are all randomly created from known probabilistic distributions. The probabilistic values of completion time within a promised date and of total cost under a given budget are also considered as response values for statistical optimisation. As is already known, the solutions found using RSM are solutions that are optimised for a single objective only. In the case of multiple objectives, actions for further optimisation are necessitated within a limited project budget, restricted completion time, allowable total cost probability, and acceptable completion time probability. Hence, after gathering response functions using RSM, an optimisation technique, mathematical programming, is used to formulate the problem in an integral fashion and solve multiple objectives under various constraints.