A generalized net model of the stochastic gradient descent and dropout algorithm with intuitionistic fuzzy evaluations

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that the deviation of the slow motion from the averaged equation, after proper rescaling, converges to a stochastic process with Gaussian inputs. This indicates that the slow motion can be approximated in the weak sense by a standard perturbed gradient flow or the continuous-time stochastic gradient descent algorithm that solves the optimization problem for a composition of two functions. As an application, the perturbed compositional gradient flow corresponds to the diffusion limit of the Stochastic Composite Gradient Descent (SCGD) algorithm for minimizing a composition of two expected-value functions in the optimization literatures. For the strongly convex case, such an analysis implies that the SCGD algorithm has the same convergence time asymptotic as the classical stochastic gradient descent algorithm. Thus it validates, at the level of continuous approximation, the effectiveness of using the SCGD algorithm in the strongly convex case.

Linear prediction methods, such as least squares for regression, logistic regression and support vector machines for classification, have been extensively used in statistics and machine learning. In this paper, we study stochastic gradient descent (SGD) algorithms on regularized forms of linear prediction methods. This class of methods, related to online algorithms such as perceptron, are both efficient and very simple to implement. We obtain numerical rate of convergence for such algorithms, and discuss its implications. Experiments on text data will be provided to demonstrate numerical and statistical consequences of our theoretical findings.

Structural health monitoring (SHM) provides an economic approach which aims to enhance understanding the behavior of structures by continuously collecting data through multiple networked sensors attached to the structure. These data are then utilized to gain insight into the health of a structure and make timely and economic decisions about its maintenance. The generated SHM sensing data are non-stationary and exist in a correlated multi-way form which makes the batch/off-line learning and standard two-way matrix analysis unable to capture all of these correlations and relationships. In this sense, the online tensor data analysis has become an essential tool for capturing underlying structures in higher-order datasets stored in a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . The CANDECOMP/PARAFAC (CP) decomposition has been extensively studied and applied to approximate X by N loading matrices $$A(1),\ldots ,A(N)$$ A ( 1 ) , … , A ( N ) where N represents the order of the tensor. We propose a novel algorithm, FP-CPD, to parallelize the CANDECOMP/PARAFAC (CP) decomposition of a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . Our approach is based on stochastic gradient descent (SGD) algorithm which allows us to parallelize the learning process, and it is very useful in online setting since it updates $${\mathcal {X}}^{t+1}$$ X t + 1 in one single step. Our SGD algorithm is augmented with Nesterov’s accelerated gradient and perturbation methods to accelerate and guarantee convergence. The experimental results using laboratory-based and real-life structural datasets indicate fast convergence and good scalability.

Stochastic differential equation (SDE) models are the foundation for pricing and hedging financial derivatives. The drift and volatility functions in SDE models are typically chosen to be algebraic functions with a small number ( < 5 ) of parameters which can be calibrated to market data. A more flexible approach is to use neural networks to model the drift and volatility functions, which provides more degrees-of-freedom to match observed market data. Training of models requires optimizing over an SDE, which is computationally challenging. For European options, we develop a fast stochastic gradient descent (SGD) algorithm for training the neural network-SDE model. Our SGD algorithm uses two independent SDE paths to obtain an unbiased estimate of the direction of steepest descent. For American options, we optimize over the corresponding Kolmogorov partial differential equation (PDE). The neural network appears as coefficient functions in the PDE. Models are trained on large datasets (many contracts), requiring either large simulations (many Monte Carlo samples for the stock price paths) or large numbers of PDEs (a PDE must be solved for each contract). Numerical results are presented for real market data including S&P 500 index options, S&P 100 index options, and single-stock American options. The neural-network-based SDE models are compared against the Black-Scholes model, the Dupire's local volatility model, and the Heston model. Models are evaluated in terms of how accurate they are at pricing out-of-sample financial derivatives, which is a core task in derivative pricing at financial institutions. Specifically, we calibrate a neural network-SDE model to market data for a financial derivative on an asset with price S t with a payoff function g ( s ) , and we then evaluate its generalization accuracy for a financial derivative on the same asset S t but with a different payoff function f ( s ) . In addition to comparing out-of-sample pricing accuracy, we evaluate the hedging performance of the neural network-SDE model.

Efficiency of large-scale learning is a hot topic in both academic and industry. The stochastic gradient descent (SGD) algorithm, and its extension mini-batch SGD, allow the model to be updated without scanning the whole data set. However, the use of approximate gradient leads to the uncertainty issue, slowing down the decreasing of objective function. Furthermore, such uncertainty may result in a high frequency of meaningless update on the model, causing a communication issue in parallel learning environment. In this work, we develop a batch-adaptive stochastic gradient descent (BA-SGD) algorithm, which can dynamically choose a proper batch size as learning proceeds. Particularly on the basis of Taylor extension and central limit theorem, it models the decrease of objective value as a Gaussian random walk game with rebound. In this game, a heuristic strategy of determining batch size is adopted to maximize the utility of each incremental sampling. By evaluation on multiple real data sets, we demonstrate that by smartly choosing the batch size, the BA-SGD not only conserves the fast convergence of SGD algorithm but also avoids too frequent model updates.

This paper considers blind deconvolution problem that to recover unknown signals fand g from their convolution signal. Non-convex optimization approach is an efficient method to get the solution, but it is a challenge to find the exact solution for a non-convex optimization problem. Existing work provides a full gradient descent (GD) method converging to the global minimum from a proper initialization. However, GD algorithm is not computationally efficient. In this paper, we design the first stochastic gradient descent (SGD) algorithm that converges linearly to the exact solution. We also design a Kaczmarz algorithm which adapts the step size of SGD algorithm. It also has the linear convergence and is more computationally efficient. Finally, we analyze the global geometry of the objective function. Although the function is non-convex, its expectation has a good geometry that every local minimum is also a global optimal point. Our numerical experiments demonstrate that both SGD and Kaczmarz algorithms are more computationally efficient and can converge to the global minimum even without a proper initialization.

We provide parallel algorithms for large-scale matrix completion on problems with millions of rows, millions of columns, and billions of revealed entries. We focus on in-memory algorithms that run either in a shared-memory environment on a powerful compute node or in a shared-nothing environment on a small cluster of commodity nodes; even very large problems can be handled effectively in these settings. Our ASGD, DSGD-MR, DSGD++, and CSGD algorithms are novel variants of the popular stochastic gradient descent (SGD) algorithm, with the latter three algorithms based on a new "stratified SGD" approach. All of the algorithms are cache-friendly and exploit thread-level parallelism, in-memory processing, and asynchronous communication. We investigate the performance of both new and existing algorithms via a theoretical complexity analysis and a set of large-scale experiments. The results show that CSGD is more scalable, and up to 60 % faster, than the best-performing alternative method in the shared-memory setting. DSGD++ is superior in terms of overall runtime, memory consumption, and scalability in the shared-nothing setting. For example, DSGD++ can solve a difficult matrix completion problem on a high-variance matrix with 10M rows, 1M columns, and 10B revealed entries in around 40 min on 16 compute nodes. In general, algorithms based on SGD appear to perform better than algorithms based on alternating minimizations, such as the PALS and DALS alternating least-squares algorithms.

IERUS Technologies, Inc. and the University of Alabama in Huntsville have partnered to perform characterization and development of algorithms and hardware for adaptive optics. To date the algorithm work has focused on implementation of the stochastic parallel gradient descent (SPGD) algorithm. SPGD is a metric-based approach in which a scalar metric is optimized by taking random perturbative steps for many actuators simultaneously. This approach scales to systems with a large number of actuators while maintaining bandwidth, while conventional methods are negatively impacted by the very large matrix multiplications that are required. The metric approach enables the use of higher speed sensors with fewer (or even a single) sensing element(s), enabling a higher control bandwidth. Furthermore, the SPGD algorithm is model-free, and thus is not strongly impacted by the presence of nonlinearities which degrade the performance of conventional phase reconstruction methods. Finally, for high energy laser applications, SPGD can be performed using the primary laser beam without the need for an additional beacon laser. The conventional SPGD algorithm was modified to use an adaptive gain to improve convergence while maintaining low steady state error. Results from laboratory experiments using phase plates as atmosphere surrogates will be presented, demonstrating areas in which the adaptive gain yields better performance and areas which require further investigation.

Owing to the restriction of the stimulated Brillouin scattering (SBS) and other nonlinear effects, the output power of the single frequency amplifier is ar a level of about hundred watts. Multi-wavelength and non-single frequency seed laser will mitigate the SBS and improve the output power of the fiber laser amplifier. The coherent beam combining of the non-single frequency and/or multi-wavelength laser will greatly increase the total output power of the coherent beam combining as compared with the coherent beam combining of a single frequency laser. Based on the stochastic parallel gradient descent (SPGD) algorithm, coherent combining of four channel four-wavelength lasers is realized. In the close loop state, the mean power of the main-lobe is four times higher than that of the open-loop, approaching 75% of its ideal value. The coherent combining of the multi-wavelength and non-single frequency laser is demonstrated and an approach to the high power coherent beam combining is provided.

Coherent beam combining (CBC) of fiber laser array is a promising way to achieve high output power. Phase control is one key point to implement CBC. Appropriate feedback structures should be established to achieve phase control. Most feedback structures of CBC are established after the lasers emit to free space and consist of a set of lenses or mirrors. Those optical elements in free space may hinder array size and integration. In this paper, we demonstrated an all-fiber structured CBC method with distributed phase-locking. By adding an all-fiber measurement loop beside the main laser chain, the phase of main laser chain is appended to the measuring loop. Phases of each main laser chain are locked indirectly though the measurement loops by using stochastic parallel gradient descent (SPGD) algorithm. The principle of distributed phase-locking is also illustrated. Corresponding simulations are carried out and two-channel fiber lasers are coherently combined by this method. The experimental results show that the structure can achieve phase-locking effectively. Stable and distinct interference fringe is observed. Additionally, the structure proposed in this paper is straightforwardly building and expanding.

Clickbait has been considered a problem in the modern age of technology, especially in Indonesia. Various attempts have been made to research clickbait detection, however, researches which use Indonesian data are relatively scarce compared to other languages such as English because the availability of Indonesian datasets are lacking. Backpropagation Neural Network was used in clickbait detection on article's title and has achieved quite good accuracy result, however there is still a chance to improve the accuracy. This paper shows the results of using a modified backpropagation neural network algorithm to detect clickbait using article titles when compared to the standard algorithm. The research compares the results of standard stochastic gradient descent algorithm, mini-batch gradient descent algorithm, and a version of stochastic gradient descent with Adam optimizer and three hidden layers. The results show that using Adam optimizer and three hidden layers in stochastic gradient descent algorithm significantly improves the results compared to the standard architecture. The modified algorithm shows a precision score of 78% and a recall and F1 score of 76%, where the standard algorithm has a precision score of 67% and a recall and F1 score of 66%. The resulting algorithm is then implemented to a desktop application, which is considered easy to use.

A recommender system (RS) relying on latent factor analysis usually adopts stochastic gradient descent (SGD) as its learning algorithm. However, owing to its serial mechanism, an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems. Aiming at addressing this issue, this study proposes a momentum-incorporated parallel stochastic gradient descent (MPSGD) algorithm, whose main idea is two-fold: a) implementing parallelization via a novel data-splitting strategy, and b) accelerating convergence rate by integrating momentum effects into its training process. With it, an MPSGD-based latent factor (MLF) model is achieved, which is capable of performing efficient and high-quality recommendations. Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm, an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

Latent factor analysis (LFA) is highly efficient in knowledge discovery from high-dimensional and sparse (HiDS) matrices frequently encountered in big data and web service related applications. A stochastic gradient descent (SGD) algorithm is commonly adopted as a learning algorithm for LFA owing to its high efficiency. However, its sequential nature makes it less scalable when processing large-scale data. Although an alternating SGD algorithm decouples an LFA process to achieve parallelization, its performance relies on its hyper-parameter selection that is highly expensive to tune. To address it, this paper presents three adaptive alternating SGD algorithms, thus leading to three Parallel Adaptive LFA (PAL) models for LFA on large-scale HiDS matrices. Experimental studies on HiDS matrices from industrial service applications show that the proposed PAL models perform significantly better than existing ones in terms of both convergence rate and computational efficiency, as well as achieve competitive prediction accuracy for missing data.

Interval-valued intuitionistic fuzzy soft sets and their properties

A high-speed phase calibration method for large-scale silicon-based optical phased arrays (OPAs) is proposed, integrating the Hadamard parallel gradient descent (HPGD) algorithm and a dedicated hardware architecture. Deterministic, orthogonal Hadamard matrices generate perturbation sequences, enabling efficient optimization. The architecture consists of an FPGA, a DSP, and a high-speed avalanche photodiode (APD), employing the APD to acquire objective functions. Simulations show that, compared with the stochastic parallel gradient descent algorithm, HPGD reduces convergence iterations for a 512-element OPA from 4929 to 2285, and a variant with adaptive gain (adaHPGD) further reduces iterations to 1480 (70.0% reduction). For OPAs with 1024 to 4096 elements, HPGD reduces iterations by 31.8% to 41.3%. Experimentally, on a 512-element OPA, each iteration takes 6.7 ms, with a convergence time of 12.8 s. After calibration, the beam achieves a peak sidelobe ratio of 12.93 dB and a full width at half maximum of 0.110°, matching simulations.