Adaptive Alternating Stochastic Gradient Descent Algorithms for Large-Scale Latent Factor Analysis

A latent factor analysis (LFA) model can efficiently address a high-dimensional and sparse (HiDS) matrix with a stochastic gradient descent (SGD) algorithm. However, an SGD-based LFA model's performance depends heavily on its hyper-parameters. The popular method is based on grid-search, which fails because of expensive computation and time-consuming. Aiming at implementing a hyper-parameter-free LFA model, this study proposes an adaptive moment estimation-incorporated particle swarm optimization (Adam-PSO) algorithm that efficiently addresses the premature issues in a PSO algorithm. With achieved the hyper-parameter adaptation in an SGD-based LFA model, an Adam-PSO-based LFA (APL) model possessing hyper-parameter-free training is further implemented. Empirical studies on four HiDS matrices indicate that compared with state-of-the-art models with hyper-parameter adaptation settings, an APL model achieves the most efficient hyper-parameter-free training and highly competitive prediction accuracy for missing data of an HiDS matrix. Hence, it fits the need of real applications with high scalability and efficiency.

A recommender system (RS) relying on latent factor analysis usually adopts stochastic gradient descent (SGD) as its learning algorithm. However, owing to its serial mechanism, an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems. Aiming at addressing this issue, this study proposes a momentum-incorporated parallel stochastic gradient descent (MPSGD) algorithm, whose main idea is two-fold: a) implementing parallelization via a novel data-splitting strategy, and b) accelerating convergence rate by integrating momentum effects into its training process. With it, an MPSGD-based latent factor (MLF) model is achieved, which is capable of performing efficient and high-quality recommendations. Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm, an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

User-side Quality-of-Service (QoS) data are vital for efficient cloud service selection, while available QoS data in real applications are commonly described by a high-dimensional and Sparse (HiDS) matrix due to a) increasing users and services, and b) the impossibility of observing the full invoking mapping among users and services. A latent factor analysis (LFA) model has proven to be efficient in performing representation learning on such an HiDS QoS matrix. However, existing LFA models commonly adopt a first-order optimizer that cannot makes it approach the second-order stationary point of the learning objective, thereby resulting in accuracy loss. Aiming at addressing this issue, this study proposes to incorporate a Generalized Nesterov's Acceleration (GNA) method in to a Hessian-vector algorithm for LFA, thereby establishing a GNA-incorporated Hessian-vector-based LFA (GNHL) model with two-fold ideas: a) adopting the principle of a Hessian-vector method to acquire a proper Newton step efficiently, and b) incorporating a GNA method into its linear search for accelerating its convergence rate. Experimental results on six real QoS datasets demonstrate that a GNHL model outperforms state-of-the-art LFA models in generating highly accurate predictions for missing QoS data with low computational burden.

Linear prediction methods, such as least squares for regression, logistic regression and support vector machines for classification, have been extensively used in statistics and machine learning. In this paper, we study stochastic gradient descent (SGD) algorithms on regularized forms of linear prediction methods. This class of methods, related to online algorithms such as perceptron, are both efficient and very simple to implement. We obtain numerical rate of convergence for such algorithms, and discuss its implications. Experiments on text data will be provided to demonstrate numerical and statistical consequences of our theoretical findings.

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that the deviation of the slow motion from the averaged equation, after proper rescaling, converges to a stochastic process with Gaussian inputs. This indicates that the slow motion can be approximated in the weak sense by a standard perturbed gradient flow or the continuous-time stochastic gradient descent algorithm that solves the optimization problem for a composition of two functions. As an application, the perturbed compositional gradient flow corresponds to the diffusion limit of the Stochastic Composite Gradient Descent (SCGD) algorithm for minimizing a composition of two expected-value functions in the optimization literatures. For the strongly convex case, such an analysis implies that the SCGD algorithm has the same convergence time asymptotic as the classical stochastic gradient descent algorithm. Thus it validates, at the level of continuous approximation, the effectiveness of using the SCGD algorithm in the strongly convex case.

In the paper, we consider a stochastic gradient descent algorithm in combination with a dropout method. We used the theory of intuitionistic fuzzy sets for the assessment of the equivalence of the respective assessment units. We also consider a degree of uncertainty when the information is not enough.

Efficiency of large-scale learning is a hot topic in both academic and industry. The stochastic gradient descent (SGD) algorithm, and its extension mini-batch SGD, allow the model to be updated without scanning the whole data set. However, the use of approximate gradient leads to the uncertainty issue, slowing down the decreasing of objective function. Furthermore, such uncertainty may result in a high frequency of meaningless update on the model, causing a communication issue in parallel learning environment. In this work, we develop a batch-adaptive stochastic gradient descent (BA-SGD) algorithm, which can dynamically choose a proper batch size as learning proceeds. Particularly on the basis of Taylor extension and central limit theorem, it models the decrease of objective value as a Gaussian random walk game with rebound. In this game, a heuristic strategy of determining batch size is adopted to maximize the utility of each incremental sampling. By evaluation on multiple real data sets, we demonstrate that by smartly choosing the batch size, the BA-SGD not only conserves the fast convergence of SGD algorithm but also avoids too frequent model updates.

Structural health monitoring (SHM) provides an economic approach which aims to enhance understanding the behavior of structures by continuously collecting data through multiple networked sensors attached to the structure. These data are then utilized to gain insight into the health of a structure and make timely and economic decisions about its maintenance. The generated SHM sensing data are non-stationary and exist in a correlated multi-way form which makes the batch/off-line learning and standard two-way matrix analysis unable to capture all of these correlations and relationships. In this sense, the online tensor data analysis has become an essential tool for capturing underlying structures in higher-order datasets stored in a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . The CANDECOMP/PARAFAC (CP) decomposition has been extensively studied and applied to approximate X by N loading matrices $$A(1),\ldots ,A(N)$$ A ( 1 ) , … , A ( N ) where N represents the order of the tensor. We propose a novel algorithm, FP-CPD, to parallelize the CANDECOMP/PARAFAC (CP) decomposition of a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . Our approach is based on stochastic gradient descent (SGD) algorithm which allows us to parallelize the learning process, and it is very useful in online setting since it updates $${\mathcal {X}}^{t+1}$$ X t + 1 in one single step. Our SGD algorithm is augmented with Nesterov’s accelerated gradient and perturbation methods to accelerate and guarantee convergence. The experimental results using laboratory-based and real-life structural datasets indicate fast convergence and good scalability.

Stochastic differential equation (SDE) models are the foundation for pricing and hedging financial derivatives. The drift and volatility functions in SDE models are typically chosen to be algebraic functions with a small number ( < 5 ) of parameters which can be calibrated to market data. A more flexible approach is to use neural networks to model the drift and volatility functions, which provides more degrees-of-freedom to match observed market data. Training of models requires optimizing over an SDE, which is computationally challenging. For European options, we develop a fast stochastic gradient descent (SGD) algorithm for training the neural network-SDE model. Our SGD algorithm uses two independent SDE paths to obtain an unbiased estimate of the direction of steepest descent. For American options, we optimize over the corresponding Kolmogorov partial differential equation (PDE). The neural network appears as coefficient functions in the PDE. Models are trained on large datasets (many contracts), requiring either large simulations (many Monte Carlo samples for the stock price paths) or large numbers of PDEs (a PDE must be solved for each contract). Numerical results are presented for real market data including S&P 500 index options, S&P 100 index options, and single-stock American options. The neural-network-based SDE models are compared against the Black-Scholes model, the Dupire's local volatility model, and the Heston model. Models are evaluated in terms of how accurate they are at pricing out-of-sample financial derivatives, which is a core task in derivative pricing at financial institutions. Specifically, we calibrate a neural network-SDE model to market data for a financial derivative on an asset with price S t with a payoff function g ( s ) , and we then evaluate its generalization accuracy for a financial derivative on the same asset S t but with a different payoff function f ( s ) . In addition to comparing out-of-sample pricing accuracy, we evaluate the hedging performance of the neural network-SDE model.

This paper considers blind deconvolution problem that to recover unknown signals fand g from their convolution signal. Non-convex optimization approach is an efficient method to get the solution, but it is a challenge to find the exact solution for a non-convex optimization problem. Existing work provides a full gradient descent (GD) method converging to the global minimum from a proper initialization. However, GD algorithm is not computationally efficient. In this paper, we design the first stochastic gradient descent (SGD) algorithm that converges linearly to the exact solution. We also design a Kaczmarz algorithm which adapts the step size of SGD algorithm. It also has the linear convergence and is more computationally efficient. Finally, we analyze the global geometry of the objective function. Although the function is non-convex, its expectation has a good geometry that every local minimum is also a global optimal point. Our numerical experiments demonstrate that both SGD and Kaczmarz algorithms are more computationally efficient and can converge to the global minimum even without a proper initialization.

We provide parallel algorithms for large-scale matrix completion on problems with millions of rows, millions of columns, and billions of revealed entries. We focus on in-memory algorithms that run either in a shared-memory environment on a powerful compute node or in a shared-nothing environment on a small cluster of commodity nodes; even very large problems can be handled effectively in these settings. Our ASGD, DSGD-MR, DSGD++, and CSGD algorithms are novel variants of the popular stochastic gradient descent (SGD) algorithm, with the latter three algorithms based on a new "stratified SGD" approach. All of the algorithms are cache-friendly and exploit thread-level parallelism, in-memory processing, and asynchronous communication. We investigate the performance of both new and existing algorithms via a theoretical complexity analysis and a set of large-scale experiments. The results show that CSGD is more scalable, and up to 60 % faster, than the best-performing alternative method in the shared-memory setting. DSGD++ is superior in terms of overall runtime, memory consumption, and scalability in the shared-nothing setting. For example, DSGD++ can solve a difficult matrix completion problem on a high-variance matrix with 10M rows, 1M columns, and 10B revealed entries in around 40 min on 16 compute nodes. In general, algorithms based on SGD appear to perform better than algorithms based on alternating minimizations, such as the PALS and DALS alternating least-squares algorithms.

Latent factor (LF) models are highly effective in extracting useful knowledge from High-Dimensional and Sparse (HiDS) matrices which are commonly seen in various industrial applications. An LF model usually adopts iterative optimizers, which may consume many iterations to achieve a local optima, resulting in considerable time cost. Hence, how to accelerate the training process of an LF model becomes a highly significant issue. To address it, this work innovatively proposes a randomized latent factor (RLF) model. It incorporates the principle of randomized learning techniques for neural networks into the LF analysis on HiDS matrices to alleviate the computational burden greatly. It also extends the standard learning process for randomized neural networks in context of LF analysis to make the resulting model represent an HiDS matrix correctly. Experimental results on three HiDS matrices from industrial applications demonstrate that compared with state-of-the-art LF models, RLF is able to achieve significantly higher computational efficiency and comparable prediction accuracy for missing data. More importantly, it provides a novel, effective, and efficient approach to LF analysis on HiDS matrices.

High-dimensional and sparse (HiDS) matrices are frequently adopted to describe the complex relationships in various big data-related systems and applications. A Position-transitional Latent Factor Analysis (PLFA) model can accurately and efficiently represent an HiDS matrix. However, its involved latent factors are optimized by particle-swarm-optimization-incorporated stochastic gradient descent with the specific gradient direction step-by-step, which may cause a suboptimal solution. To address this issue, this paper proposes a <u>S</u>equential-<u>G</u>roup-<u>D</u>ifferential-<u>E</u>volution (SGDE) algorithm to refine the latent factors optimized by a PLFA model, thereby achieving a highly-accurate SGDE-PLFA model to HiDS matrices. The main idea of SGDE-PLFA is two-fold. First, it divides the obtained latent factors by PLFA into row and column sub-groups to ensure the efficient low-dimensional optimization. Second, it initializes and optimizes these sub-group latent factors with our improved Differential Evolution algorithm to search more accurate solutions nearby the suboptimum. As demonstrated by the extensive experiments on four HiDS matrices, an SGDE-PLFA model outperforms the state-of-the-art models in representing HiDS matrices.

The K-Means clustering algorithm is a widely used technique for grouping data into clusters, with applications spanning various domains. This study presents a comparative investigation into the optimization of K-Means clustering through the evaluation of different optimization algorithms. The primary focus is on enhancing the convergence speed and computational efficiency of the K-Means algorithm, with implications for diverse real-world scenarios. The research systematically examines a range of optimization techniques, including gradient descent, stochastic gradient descent, and metaheuristic algorithms such as genetic algorithms and simulated annealing. A comprehensive analysis of convergence speed, clustering quality, and computational efficiency is conducted across these algorithms. By assessing their performance on diverse datasets, the study aims to provide insights into the trade-offs between different optimization strategies and their implications for practical clustering tasks. The results reveal distinct convergence patterns, highlighting the advantages and limitations of each optimization algorithm. Gradient-based approaches demonstrate rapid convergence but susceptibility to local optima, while stochastic gradient descent and metaheuristic algorithms exhibit a balance between exploration and exploitation. The findings shed light on the interplay between optimization techniques, convergence speed, and clustering quality, offering valuable guidance for practitioners seeking to optimize K-Means clustering according to specific dataset characteristics and computational requirements. This comparative study contributes to the broader understanding of optimizing K-Means clustering algorithms and aids researchers and practitioners in selecting suitable optimization strategies for efficient and effective data clustering in real-world applications.

Risk optimization using the Chernoff bound and stochastic gradient descent