Application of gradient descent method to the sedimentary grain-size distribution fitting

Gradient descent (GD) method is used to fit the the laser grain-size distribution of the sediments with a sum of four weighted lognormal functions. The numerical results indicate that the GD method is not only easy to operate but also could effectively optimize the parameters of the fitting function with the error decreasing steadily. Meanwhile the overall fitting results are satisfactory. As a new way of data fitting, the GD method could also be used to solve other optimization problems.

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb() are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

One of the earliest and the best method to minimize a function is the classical steepest descent (SD) method. In this paper, a new modification of SD method is suggested using a new search direction, d k. The numerical results are presented based on number of iterations and CPU time. It shows that the new d k are efficient when compared to the classical SD.

Projected gradient descent method for cardinality-constrained portfolio optimization

Background: As a popular optimization algorithm, gradient descent is more prone to converge at a local minimum rather than a global one since it is highly dependent on how initial values are instantiated and an iterative process of updates, reducing its efficiency for non-convex cost functions. This limitation is of utmost importance in domains such as medical physics, where there exist many treatment parameters that need to be optimized. Purpose: We introduce the flat plane with maximum perpendicular distance method, a new optimization algorithm which identifies global minima robustly and efficiently in convex cost functions. This facilitates an initialization-free, non-iterative update method that serves as a complementary approach to classic methods. Methods: 1D and 2D convex cost functions, with both global and relative minima, were tested using the flat plane method. Further, the approach was employed to determine optimal locations for radiation beams in a medical physics problem. Results were compared to existing gradient descent methods for effectiveness assessment. Results: The flat plane algorithm found the global minimum in all the scenarios considered while the use of gradient descent would have failed to do so in many cases due to initialization issues. However, due to a less direct but highly parallelizable strategy (i.e. flat plane method), these techniques are certainly scalable for high-dimensional problems with potentially higher expense (e.g. dimensionality reduction, sampling, parallel computing). Conclusion: In low-dimensional problems, the flat plane method provides a strong alternative to gradient descent for the global optimization problem. We are yet to adapt the method for high-dimensional applications but will work in that direction to increase its applicability over more complex optimization problems

Gradient descent is a widely used paradigm for solving many optimization problems. Stochastic gradient descent performs a series of iterations to minimize a target function in order to reach a local minimum. In machine learning or data mining, this function corresponds to a decision model that is to be discovered. The gradient descent paradigm underlies many commonly used techniques in data mining and machine learning, such as neural networks, Bayesian networks, genetic algorithms, and simulated annealing. To the best of our knowledge, there has not been any work that extends the notion of privacy preservation or secure multi-party computation to gradient-descent-based techniques. In this paper, we propose a preliminary approach to enable privacy preservation in gradient descent methods in general and demonstrate its feasibility in specific gradient descent methods.

Based on an internal tidal model, the practical performances of the limited-memory BFGS (L-BFGS) method and two gradient descent (GD) methods (the normal one with Wolfe’s line search and the simplified one) are investigated computationally through a series of ideal experiments in which the open boundary conditions (OBCs) are inverted by assimilating the interior observations with the adjoint method. In the case that the observations closer to the unknown boundary are included for assimilation, the L-BFGS method performs the best. As compared with the simplified GD method, the normal one really uses less iteration to reach a satisfactory solution, but its advantage over the simplified one is much smaller than expected. In the case that only the observations that are further from the unknown boundary are assimilated, the simplified GD method performs the best instead, whereas the performances of the other two methods are not satisfactory. The advanced L-BFGS algorithm and Wolfe’s line search still need to be improved when applied to the practical cases. The simplified GD method, which is controllable and easy to implement, should be regarded seriously as a choice, especially when the classical advanced optimization techniques fail or perform poorly.

Much recent attention has been devoted to gradient descent algorithms where the steepest descent step size is replaced by a similar one from a previous iteration or gets updated only once every second step, thus forming a faster gradient descent method. For unconstrained convex quadratic optimization these methods can converge much faster than steepest descent. But the context of interest here is application to certain ill-posed inverse problems, where the steepest descent method is known to have a smoothing, regularizing effect, and where a strict optimization solution is not necessary. Specifically, in this paper we examine the effect of replacing steepest descent by a faster gradient descent algorithm in the practical context of image deblurring and denoising tasks. We also propose several highly efficient schemes for carrying out these tasks independently of the step size selection, as well as a scheme for the case where both blur and significant noise are present. In the above context there are situations where many steepest descent steps are required, thus building slowness into the solution procedure. Our general conclusion regarding gradient descent methods is that in such cases the faster gradient descent methods offer substantial advantages. In other situations where no such slowness buildup arises the steepest descent method can still be very effective.

COVID-19 is a type of an infectious disease that is caused by the new coronavirus. The spread of COVID-19 needs to be suppressed because COVID-19 can cause death, especially for sufferers with congenital diseases and a weak immune system. COVID-19 spreads through direct contact, wherein the infected individual spreads the COVID-19 virus through cough, sneeze, or close contacts. Predicting the number of COVID-19 sufferers becomes an important task in the effort to curb the spread of COVID-19. Artificial neural network (ANN) is the prediction method that delivers effective results in doing this job. Backpropagation, a type of ANN algorithm, offers predictive problem solving with good performance. However, its performance depends on the optimization method applied during the training process. In general, the optimization method in ANN is the gradient descent method, which is known to have a slow convergence rate. Meanwhile, the Fletcher–Reeves method has a faster convergence rate than the gradient descent method. Based on this hypothesis, this paper proposes a prediction model for the number of COVID-19 sufferers in Malang using the Backpropagation neural network with the Fletcher–Reeves method. The experimental results show that the Backpropagation neural network with the Fletcher–Reeves method has a better performance than the Backpropagation neural network with the gradient descent method. This is shown by the Means Square Error (MSE) resulting from the proposed method which is smaller than the MSE resulting from the Backpropagation neural network with the gradient descent method.

An optimization method of adjoint assimilation based on the cumulative gradient: An application for correction of the bottom friction coefficient in the tidal wave model

Huber Loss function is a regression type loss function, mainly used for optimization problems.In order to make the predicted value close to the true value and minimize the distance between them, the traditional loss function generally uses the gradient descent method to find the minimum value. However, the gradient descent method takes too long and consumes a lot of memory. It may not necessarily find the global optimal value, and it is easy to fall into the problem of local optimization. The data of the unconstrained function is complicated and huge, and the Huber Loss function sets the hyper-parameter δ to optimize MSE and MAE, we must iterate continuously to train the hyper-parameters to obtain more accurate parameter values, which will cause a problem of large amount of calculations. The beetle antennae search algorithm is a simple biological intelligence optimization algorithm that can provide the best solution based on huge data, with fast convergence speed, small amount of calculation, simple operation, and no need to know the specific function. In order to improve the accuracy of finding the global optimal value, this paper changed the traditional fixed-step antennae search algorithm to variable-step beetle antennae search algorithm. This article will discuss the application of the gradient descent method in the regression loss function, and then use the variable step size beetle antennae search algorithm to optimize Huber Loss function.

We present a novel hybrid algorithm for training Deep Neural Networks that combines the state-of-the-art Gradient Descent (GD) method with a Mixed Integer Linear Programming (MILP) solver, outperforming GD and variants in terms of accuracy, as well as resource and data efficiency for both regression and classification tasks. Our GD+Solver hybrid algorithm, called GDSolver, works as follows: given a DNN D as input, GDSolver invokes GD to partially train D until it gets stuck in a local minima, at which point GDSolver invokes an MILP solver to exhaustively search a region of the loss landscape around the weight assignments of D’s final layer parameters with the goal of tunnelling through and escaping the local minima. The process is repeated until desired accuracy is achieved. In our experiments, we find that GDSolver not only scales well to additional data and very large model sizes, but also outperforms all other competing methods in terms of rates of convergence and data efficiency. For regression tasks, GDSolver produced models that, on average, had 31.5% lower MSE in 48% less time, and for classification tasks on MNIST and CIFAR10, GDSolver was able to achieve the highest accuracy over all competing methods, using only 50% of the training data that GD baselines required.

We consider minimizing a class of nonconvex composite stochastic optimization problems, and deterministic optimization problems whose objective function consists of an expectation function (or an average of finitely many smooth functions) and a weakly convex but potentially nonsmooth function. And in this paper, we focus on the theoretical properties of two types of stochastic splitting methods for solving these nonconvex optimization problems: stochastic alternating direction method of multipliers and stochastic proximal gradient descent. In particular, several inexact versions of these two types of splitting methods are studied. At each iteration, the proposed schemes inexactly solve their subproblems by using relative error criteria instead of exogenous and diminishing error rules, which allows our approaches to handle some complex regularized problems in statistics and machine learning. Under mild conditions, we obtain the convergence of the schemes and their computational complexity related to the evaluations on the component gradient of the smooth function, and find that some conclusions of their exact counterparts can be recovered.

PurposeThe proportional‐integral‐derivative (PID) controller has been a practical application in industry due to its simple architecture, being easily designed and its parameter tuning without complicated computation. However, the traditional PID controller usually needs some manual retuning before being used for practical application in industry. The purpose of this paper is to propose an auto‐tuning PID controller (ATPIDC) which can automatically tune the controller parameters based on the gradient descent method and the Lyapunov stability theorem. Finally, a field‐programmable gate array (FPGA) chip is adopted to implement the proposed ATPIDC scheme for possible low‐cost and high‐performance industrial applications, and it is applied to a DC servomotor to show its effectiveness.Design/methodology/approachTo ensure the stability of the intelligent control system, a compensator usually should be designed. The most frequently used compensator is designed as a sliding‐mode control, which results in substantial chattering in the control effort. To tackle this problem, the proposed ATPIDC system is composed of a PID controller and a fuzzy compensator. The PID controller can automatically tune the gain factors of the controller gains based on the gradient descent method, and the fuzzy compensator is utilized to eliminate approximation error based on the Lyapunov stability theorem. The proposed fuzzy compensator not only can remove the chattering phenomena of conventional sliding‐mode control completely, but also can guarantee the stability of the closed‐loop system.FindingsThe proposed ATPIDC system is applied to a DC servomotor on a FPGA chip. The hardware implementation of the ATPIDC scheme is developed in a real‐time mode. Using the FPGA to implement, the ATPIDC system can achieve the characteristics of small size, fast execution speed and less memory. A comparison among the fuzzy sliding‐mode control, adaptive robust PID control and the proposed ATPIDC is made. Experimental results verify a better position tracking response can be achieved by the proposed ATPIDC method after control parameters training.Originality/valueThe proposed ATPIDC approach is interesting for the design of an intelligent control scheme. An on‐line parameter training methodology, using the gradient descent method and the Lyapunov stability theorem, is proposed to increase the learning capability. The experimental results verify the system stabilization, favorable tracking performance and no chattering phenomena can be achieved by using the proposed ATPIDC system. Also, the proposed ATPIDC methodology can be easily extended to other motors.